[jira] [Updated] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23443:
-
Attachment: HIVE-23443.3.patch
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch,
> HIVE-23443.3.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108798#comment-17108798
]
Prasanth Jayachandran commented on HIVE-23443:
--
[~pgaref] non-finishable to finishable is not a problem. But there is concern
in the line that you pinged in PR that double/multiple addition could be
possible with pre-emption queue and I was able to unit test it. Could you look
at the diff in PR again?
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?focusedWorklogId=433984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433984
]
ASF GitHub Bot logged work on HIVE-23443:
-
Author: ASF GitHub Bot
Created on: 16/May/20 01:19
Start Date: 16/May/20 01:19
Worklog Time Spent: 10m
Work Description: prasanthj commented on a change in pull request #1012:
URL: https://github.com/apache/hive/pull/1012#discussion_r426102275
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -884,10 +885,20 @@ private void finishableStateUpdated(TaskWrapper
taskWrapper, boolean newFinishab
taskWrapper.updateCanFinishForPriority(newFinishableState);
forceReinsertIntoQueue(taskWrapper, isRemoved);
} else {
-taskWrapper.updateCanFinishForPriority(newFinishableState);
-if (!newFinishableState && !taskWrapper.isInPreemptionQueue()) {
- // No need to check guaranteed here; if it was false we would
already be in the queue.
+// if speculative task, any finishable state change should re-order
the queue as speculative tasks are always
+// not-guaranteed (re-order helps put non-finishable's ahead of
finishable)
+if (!taskWrapper.isGuaranteed()) {
+ removeFromPreemptionQueue(taskWrapper);
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
addToPreemptionQueue(taskWrapper);
+} else {
+ // if guaranteed task, if the finishable state changed to
non-finishable and if the task doesn't exist
+ // pre-emption queue, then add it so that it becomes candidate to
kill
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
Review comment:
Non-finishable -> Finishable does not have to be pre-emption queue. This
could be in wait queue (if not capacity) or taken by executor to run both of
which are fine.
You brought up good point, we may be adding the same task fragment to
pre-emption queue twice. I will add a "if not exists" check when adding to
pre-emption queue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 433984)
Time Spent: 40m (was: 0.5h)
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?focusedWorklogId=433983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433983
]
ASF GitHub Bot logged work on HIVE-23443:
-
Author: ASF GitHub Bot
Created on: 16/May/20 01:18
Start Date: 16/May/20 01:18
Worklog Time Spent: 10m
Work Description: prasanthj commented on a change in pull request #1012:
URL: https://github.com/apache/hive/pull/1012#discussion_r426102275
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -884,10 +885,20 @@ private void finishableStateUpdated(TaskWrapper
taskWrapper, boolean newFinishab
taskWrapper.updateCanFinishForPriority(newFinishableState);
forceReinsertIntoQueue(taskWrapper, isRemoved);
} else {
-taskWrapper.updateCanFinishForPriority(newFinishableState);
-if (!newFinishableState && !taskWrapper.isInPreemptionQueue()) {
- // No need to check guaranteed here; if it was false we would
already be in the queue.
+// if speculative task, any finishable state change should re-order
the queue as speculative tasks are always
+// not-guaranteed (re-order helps put non-finishable's ahead of
finishable)
+if (!taskWrapper.isGuaranteed()) {
+ removeFromPreemptionQueue(taskWrapper);
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
addToPreemptionQueue(taskWrapper);
+} else {
+ // if guaranteed task, if the finishable state changed to
non-finishable and if the task doesn't exist
+ // pre-emption queue, then add it so that it becomes candidate to
kill
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
Review comment:
Non-finishable -> Finishable does not have to be pre-emption queue. This
could be in wait queue (if not capacity) or taken by executor to run both of
which are fine.
You brought up good point, we may be adding the same task fragment to
pre-emption queue twice. I will add a if not exists check when adding to
pre-emption queue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 433983)
Time Spent: 0.5h (was: 20m)
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?focusedWorklogId=433982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433982
]
ASF GitHub Bot logged work on HIVE-23443:
-
Author: ASF GitHub Bot
Created on: 16/May/20 01:10
Start Date: 16/May/20 01:10
Worklog Time Spent: 10m
Work Description: prasanthj commented on a change in pull request #1012:
URL: https://github.com/apache/hive/pull/1012#discussion_r426102275
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -884,10 +885,20 @@ private void finishableStateUpdated(TaskWrapper
taskWrapper, boolean newFinishab
taskWrapper.updateCanFinishForPriority(newFinishableState);
forceReinsertIntoQueue(taskWrapper, isRemoved);
} else {
-taskWrapper.updateCanFinishForPriority(newFinishableState);
-if (!newFinishableState && !taskWrapper.isInPreemptionQueue()) {
- // No need to check guaranteed here; if it was false we would
already be in the queue.
+// if speculative task, any finishable state change should re-order
the queue as speculative tasks are always
+// not-guaranteed (re-order helps put non-finishable's ahead of
finishable)
+if (!taskWrapper.isGuaranteed()) {
+ removeFromPreemptionQueue(taskWrapper);
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
addToPreemptionQueue(taskWrapper);
+} else {
+ // if guaranteed task, if the finishable state changed to
non-finishable and if the task doesn't exist
+ // pre-emption queue, then add it so that it becomes candidate to
kill
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
Review comment:
Non-finishable -> Finishable does not have to be pre-emption queue. This
could be in wait queue (if not capacity) or taken by executor to run both which
are fine.
You brought up good point, we may be adding the same task fragment to
pre-emption queue twice. I will add a if not exists check when adding to
pre-emption queue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 433982)
Time Spent: 20m (was: 10m)
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23476) LLAP: Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal Vijayaraghavan updated HIVE-23476: Summary: LLAP: Preallocate arenas for mmap case as well (was: [LLAP] Preallocate arenas for mmap case as well) > LLAP: Preallocate arenas for mmap case as well > -- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108780#comment-17108780
]
Gopal Vijayaraghavan commented on HIVE-23477:
-
LGTM - +1
The mmap() is much safer in the startup, where it cannot be interrupted. And
that's done in HIVE-23476
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
>
[jira] [Commented] (HIVE-23476) [LLAP] Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108779#comment-17108779 ] Gopal Vijayaraghavan commented on HIVE-23476: - I think the prealloc discussion was originally about posix_fallocate(), but this is fine considering we're mmap'ing and truncating without actually taking up disk space. LGTM - +1 > [LLAP] Preallocate arenas for mmap case as well > --- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23477) LLAP : mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal Vijayaraghavan updated HIVE-23477:
Summary: LLAP : mmap allocation interruptions fails to notify other threads
(was: [LLAP] mmap allocation interruptions fails to notify other threads)
> LLAP : mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation if mmap is enabled. If query
> fragment is interrupted at the time of arena allocation,
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
>
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Description:
BuddyAllocator always uses lazy allocation if mmap is enabled. If query
fragment is interrupted at the time of arena allocation
ClosedByInterruptionException is thrown. This exception artificially triggers
allocator OutOfMemoryError and fails to notify other threads waiting to
allocate arenas.
{code:java}
2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
trying to allocate memory mapped arena
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
at
org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
at
org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
at
org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
at
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Description:
BuddyAllocator always uses lazy allocation if mmap is enabled. If query
fragment is interrupted at the time of arena allocation,
ClosedByInterruptionException is thrown. This exception artificially triggers
allocator OutOfMemoryError and fails to notify other threads waiting to
allocate arenas.
{code:java}
2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
trying to allocate memory mapped arena
java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
at
org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
at
org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
at
org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
at
org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
at
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
at
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
at
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
at
org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
at
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
at
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
at
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
at
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
[jira] [Commented] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108776#comment-17108776
]
Prasanth Jayachandran commented on HIVE-23477:
--
[~ashutoshc] / [~gopalv] can you please help review this change?
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Attachment: (was: HIVE-23476.1.patch)
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Attachment: HIVE-23477.1.patch
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23477.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Work logged] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[ https://issues.apache.org/jira/browse/HIVE-23477?focusedWorklogId=433976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433976 ] ASF GitHub Bot logged work on HIVE-23477: - Author: ASF GitHub Bot Created on: 15/May/20 23:54 Start Date: 15/May/20 23:54 Worklog Time Spent: 10m Work Description: prasanthj opened a new pull request #1020: URL: https://github.com/apache/hive/pull/1020 BuddyAllocator always uses lazy allocation is mmap is enabled. If query fragment is interrupted at the time of arena allocation ClosedByInterruptionException is thrown. This exception artificially triggers allocator OutOfMemoryError and fails to notify other threads waiting to allocate arenas. ``` 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed trying to allocate memory mapped arena java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970) at org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867) at org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69) at org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900) at org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458) at org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884) at org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740) at org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330) at org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257) at org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216) at org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49) at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343) at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238) at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160) at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50) at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145) at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111) at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156) at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82) at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703) at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662) at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150) at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-23477:
--
Labels: pull-request-available (was: )
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23476.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
>
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Status: Patch Available (was: Open)
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23476.1.patch
>
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
>
[jira] [Updated] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-23477:
-
Attachment: HIVE-23476.1.patch
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23476.1.patch
>
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
>
[jira] [Commented] (HIVE-23476) [LLAP] Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108739#comment-17108739 ] Prasanth Jayachandran commented on HIVE-23476: -- [~hashutosh]/[~gopalv] can you please review the change? > [LLAP] Preallocate arenas for mmap case as well > --- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23476) [LLAP] Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-23476: - Status: Patch Available (was: Open) > [LLAP] Preallocate arenas for mmap case as well > --- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23476) [LLAP] Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-23476: - Attachment: HIVE-23476.1.patch > [LLAP] Preallocate arenas for mmap case as well > --- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-23476.1.patch > > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23479) Avoid regenerating JdbcSchema for every table in a query
[
https://issues.apache.org/jira/browse/HIVE-23479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108721#comment-17108721
]
Stamatis Zampetakis commented on HIVE-23479:
Credits for this discovery go to [~kgyrtkirk]! :)
> Avoid regenerating JdbcSchema for every table in a query
>
>
> Key: HIVE-23479
> URL: https://issues.apache.org/jira/browse/HIVE-23479
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> Currently {{CalcitePlanner}} generates a complete {{JdbcSchema}} for every
> {{JdbcTable}} in the query.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3174
> This wastes some resources since every call to {{JdbcSchema#getTable}} needs
> to communicate with the database to bring back the tables belonging to the
> schema. Moreover, the fact that a schema is created during planning is
> counter-intuitive since in principle the schema shouldn't change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108716#comment-17108716 ] John Sherman commented on HIVE-23354: - I've created https://issues.apache.org/jira/browse/HIVE-23478 for the spurious test failure. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, HIVE-23354.6.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[
https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108715#comment-17108715
]
John Sherman commented on HIVE-23478:
-
I've attempted to fix the issue by making the database unique. I've changed the
database used from
{code:java}
db~!@#$%^&*(),<>{code}
to
{code:java}
db~!@@#$%^&*(),<>{code}
(I've added an extra @)
> Fix flaky special_character_in_tabnames_quotes_1 test
> -
>
> Key: HIVE-23478
> URL: https://issues.apache.org/jira/browse/HIVE-23478
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23478.1.patch
>
>
> While testing https://issues.apache.org/jira/browse/HIVE-23354
> special_character_in_tabnames_quotes_1 failed. Searching for the test, it
> seems other patches have also had failures. I noticed that
> special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1
> use the same database/table names. I suspect this is responsible for some of
> the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Attachment: HIVE-23478.1.patch > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Status: Patch Available (was: Open) > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-23478: --- > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23477) [LLAP] mmap allocation interruptions fails to notify other threads
[
https://issues.apache.org/jira/browse/HIVE-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-23477:
> [LLAP] mmap allocation interruptions fails to notify other threads
> --
>
> Key: HIVE-23477
> URL: https://issues.apache.org/jira/browse/HIVE-23477
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
>
> BuddyAllocator always uses lazy allocation is mmap is enabled. If query
> fragment is interrupted at the time of arena allocation
> ClosedByInterruptionException is thrown. This exception artificially triggers
> allocator OutOfMemoryError and fails to notify other threads waiting to
> allocate arenas.
> {code:java}
> 2020-05-15 00:03:23.254 WARN [TezTR-128417_1_3_1_1_0] LlapIoImpl: Failed
> trying to allocate memory mapped arena
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:970)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.preallocateArenaBuffer(BuddyAllocator.java:867)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.access$1100(BuddyAllocator.java:69)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.init(BuddyAllocator.java:900)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.allocateWithExpand(BuddyAllocator.java:1458)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator$Arena.access$800(BuddyAllocator.java:884)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateWithExpand(BuddyAllocator.java:740)
> at
> org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:330)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.wrapBbForFile(MetadataCache.java:257)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:216)
> at
> org.apache.hadoop.hive.llap.io.metadata.MetadataCache.putFileMetadata(MetadataCache.java:49)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.readSplitFooter(VectorizedParquetRecordReader.java:343)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:238)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:160)
> at
> org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
> at
> org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:427)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:145)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:111)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:156)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:82)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:703)
> at
> org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:662)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:150)
> at
> org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:114)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:532)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:178)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
> at
[jira] [Updated] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[ https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23440: -- Attachment: HIVE-23440.04.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 4 > --- > > Key: HIVE-23440 > URL: https://issues.apache.org/jira/browse/HIVE-23440 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch, > HIVE-23440.03.patch, HIVE-23440.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[ https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23440: -- Attachment: HIVE-23440.04.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 4 > --- > > Key: HIVE-23440 > URL: https://issues.apache.org/jira/browse/HIVE-23440 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch, > HIVE-23440.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[ https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23440: -- Attachment: (was: HIVE-23440.04.patch) > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 4 > --- > > Key: HIVE-23440 > URL: https://issues.apache.org/jira/browse/HIVE-23440 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch, > HIVE-23440.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[ https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23440: -- Attachment: (was: HIVE-23440.04.patch) > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 4 > --- > > Key: HIVE-23440 > URL: https://issues.apache.org/jira/browse/HIVE-23440 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch, > HIVE-23440.03.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23476) [LLAP] Preallocate arenas for mmap case as well
[ https://issues.apache.org/jira/browse/HIVE-23476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-23476: > [LLAP] Preallocate arenas for mmap case as well > --- > > Key: HIVE-23476 > URL: https://issues.apache.org/jira/browse/HIVE-23476 > Project: Hive > Issue Type: Bug >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > BuddyAllocator pre-allocation of arenas does not happen for mmap cache case. > Since we are not filling up the mmap'ed buffers the upfront allocations in > constructor is cheap. This can avoid lock free allocation of arenas later in > the code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Status: Open (was: Patch Available) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Attachment: hive-23361.5.patch Status: Patch Available (was: Open) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Attachment: (was: hive-23361.5.patch) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23396) Many fixes and improvements to stabilize tests
[
https://issues.apache.org/jira/browse/HIVE-23396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23396:
Attachment: HIVE-23396.02.patch
> Many fixes and improvements to stabilize tests
> --
>
> Key: HIVE-23396
> URL: https://issues.apache.org/jira/browse/HIVE-23396
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HHIVE-23396.01.patch, HIVE-23396.01.patch,
> HIVE-23396.02.patch, HIVE-23396.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> sometimes it seems to fail ; there were also some derby exceptions earlier
> the test itself has a @Before annotated setUp ; which starts a metastore - I
> think this should be done only once ; so @BeforeClass might be a better
> option...
> {code}
> org.apache.hadoop.hive.metastore.api.NoSuchObjectException: database
> hive.default
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:719)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_252]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_252]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_252]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy41.getDatabase(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2076)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2324)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_252]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_252]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_252]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy43.create_table_req(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16497)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16481)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_252]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_252]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_252]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_252]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
> 2020-05-06T17:04:30,243 ERROR [pool-9-thread-3]
[jira] [Updated] (HIVE-23314) Upgrade to Kudu 1.12
[ https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23314: Attachment: HIVE-23314.01.patch > Upgrade to Kudu 1.12 > > > Key: HIVE-23314 > URL: https://issues.apache.org/jira/browse/HIVE-23314 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23314.01.patch, HIVE-23314.01.patch > > > we need KUDU-3044 because it could cause random failures... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23460:
Attachment: HIVE-23460.02.patch
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch,
> HIVE-23460.02.patch, HIVE-23460.02.patch, HIVE-23460.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[ https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23234: -- Attachment: HIVE-23234.6.patch > Optimize TxnHandler::allocateTableWriteIds > -- > > Key: HIVE-23234 > URL: https://issues.apache.org/jira/browse/HIVE-23234 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch, > HIVE-23234.3.patch, HIVE-23234.4.patch, HIVE-23234.5.patch, > HIVE-23234.5.patch, HIVE-23234.5.patch, HIVE-23234.6.patch, > HIVE-23234.6.patch, HIVE-23234.6.patch > > > Table write id allocation should be examined and optimized. One low hanging > fruit is batching all the PreparedStatement inserts, but there might be other > opportunities as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[
https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108647#comment-17108647
]
Hive QA commented on HIVE-23314:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003016/HIVE-23314.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17271 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_quotes_1]
(batchId=48)
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=205)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesWrite
(batchId=218)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22361/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22361/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22361/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003016 - PreCommit-HIVE-Build
> Upgrade to Kudu 1.12
>
>
> Key: HIVE-23314
> URL: https://issues.apache.org/jira/browse/HIVE-23314
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23314.01.patch
>
>
> we need KUDU-3044 because it could cause random failures...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108600#comment-17108600
]
Hive QA commented on HIVE-23460:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003015/HIVE-23460.02.patch
{color:green}SUCCESS:{color} +1 due to 39 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=225)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_quotes_1]
(batchId=48)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22360/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22360/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22360/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003015 - PreCommit-HIVE-Build
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch,
> HIVE-23460.02.patch, HIVE-23460.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work stopped] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23361 stopped by Simhadri G. - > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Status: In Progress (was: Patch Available) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23361) Optimising privilege synchroniser
[ https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23361: -- Attachment: hive-23361.5.patch Status: Patch Available (was: Open) > Optimising privilege synchroniser > - > > Key: HIVE-23361 > URL: https://issues.apache.org/jira/browse/HIVE-23361 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: hive-23361.1.patch, hive-23361.2.patch, > hive-23361.3.patch, hive-23361.4.patch, hive-23361.5.patch, hive-23361.patch, > hive-23361.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > This patch store the privilege information as bit string. This is done to > reduce the size of the tbl_col_privs tables in metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108592#comment-17108592
]
Hive QA commented on HIVE-23460:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
35s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} accumulo-handler in master has 20 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
59s{color} | {color:blue} itests/util in master has 54 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} itests/util: The patch generated 0 new + 63
unchanged - 7 fixed = 63 total (was 70) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22360/dev-support/hive-personality.sh
|
| git revision | master / a2b3b36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql accumulo-handler itests itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22360/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch,
> HIVE-23460.02.patch, HIVE-23460.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --!
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108588#comment-17108588 ] Kishen Das commented on HIVE-21637: --- [~adeshrao] For the server side implementation, I would be creating more subtasks. Will appreciate it, if you can help with some of those subtasks. Will reach out to you, once those subtask Jiras are created. > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das updated HIVE-21637: -- Status: In Progress (was: Patch Available) > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108551#comment-17108551 ] Kishen Das commented on HIVE-21637: --- [~adeshrao] I have started working on this. I would do the API changes first with no-op on server side and later will do implementation on server side to provide consistency. Cc [~thejas]. > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108550#comment-17108550 ] Thejas Nair commented on HIVE-21637: [~adeshrao] [~kishendas] has started looking into this actively since last week. Can you please co-ordinate with him ? > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishen Das reassigned HIVE-21637: - Assignee: Kishen Das (was: Daniel Dai) > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Kishen Das >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[
https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108535#comment-17108535
]
Hive QA commented on HIVE-23234:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003013/HIVE-23234.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17272 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=128)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22359/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22359/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22359/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003013 - PreCommit-HIVE-Build
> Optimize TxnHandler::allocateTableWriteIds
> --
>
> Key: HIVE-23234
> URL: https://issues.apache.org/jira/browse/HIVE-23234
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23234.1.patch, HIVE-23234.2.patch,
> HIVE-23234.3.patch, HIVE-23234.4.patch, HIVE-23234.5.patch,
> HIVE-23234.5.patch, HIVE-23234.5.patch, HIVE-23234.6.patch, HIVE-23234.6.patch
>
>
> Table write id allocation should be examined and optimized. One low hanging
> fruit is batching all the PreparedStatement inserts, but there might be other
> opportunities as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108526#comment-17108526 ] Daniel Dai commented on HIVE-21637: --- Hi, [~adeshrao], thanks for your interest on this. I am not working on it. I am happy if you can take over, but before that, can you first check with [~thejas]? > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108523#comment-17108523
]
Panagiotis Garefalakis commented on HIVE-23443:
---
Hey [~prasanth_j] latest changes LGTM -- my only concern is if there can be a
case where we have a Guaranteed task that changes from non-finishable to
finishable and is only part of the preemptionQueue -- in that case the task
will be left hanging. It seems that neither the older or the latest changes
take care of that scenario.
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-23443:
--
Labels: pull-request-available (was: )
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23443) LLAP speculative task pre-emption seems to be not working
[
https://issues.apache.org/jira/browse/HIVE-23443?focusedWorklogId=433819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433819
]
ASF GitHub Bot logged work on HIVE-23443:
-
Author: ASF GitHub Bot
Created on: 15/May/20 18:11
Start Date: 15/May/20 18:11
Worklog Time Spent: 10m
Work Description: pgaref commented on a change in pull request #1012:
URL: https://github.com/apache/hive/pull/1012#discussion_r425967771
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
##
@@ -884,10 +885,20 @@ private void finishableStateUpdated(TaskWrapper
taskWrapper, boolean newFinishab
taskWrapper.updateCanFinishForPriority(newFinishableState);
forceReinsertIntoQueue(taskWrapper, isRemoved);
} else {
-taskWrapper.updateCanFinishForPriority(newFinishableState);
-if (!newFinishableState && !taskWrapper.isInPreemptionQueue()) {
- // No need to check guaranteed here; if it was false we would
already be in the queue.
+// if speculative task, any finishable state change should re-order
the queue as speculative tasks are always
+// not-guaranteed (re-order helps put non-finishable's ahead of
finishable)
+if (!taskWrapper.isGuaranteed()) {
+ removeFromPreemptionQueue(taskWrapper);
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
addToPreemptionQueue(taskWrapper);
+} else {
+ // if guaranteed task, if the finishable state changed to
non-finishable and if the task doesn't exist
+ // pre-emption queue, then add it so that it becomes candidate to
kill
+ taskWrapper.updateCanFinishForPriority(newFinishableState);
Review comment:
Can there be a case where we have a Guaranteed task that changes from
non-finishable to finishable and is only part of the preemptionQueue?
Under that scenario our code (and the old code) would remove it from
preemptionQ and it would not be part of any other Q.
From the code below it seems that this can indeed happen:
https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L776
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 433819)
Remaining Estimate: 0h
Time Spent: 10m
> LLAP speculative task pre-emption seems to be not working
> -
>
> Key: HIVE-23443
> URL: https://issues.apache.org/jira/browse/HIVE-23443
> Project: Hive
> Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-23443.1.patch, HIVE-23443.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I think after HIVE-23210 we are getting a stable sort order and it is causing
> pre-emption to not work in certain cases.
> {code:java}
> "attempt_1589167813851__119_01_08_0
> (hive_20200511055921_89598f09-19f1-4969-ab7a-82e2dd796273-119/Map 1, started
> at 2020-05-11 05:59:22, in preemption queue, can finish)",
> "attempt_1589167813851_0008_84_01_08_1
> (hive_20200511055928_7ae29ca3-e67d-4d1f-b193-05651023b503-84/Map 1, started
> at 2020-05-11 06:00:23, in preemption queue, can finish)" {code}
> Scheduler only peek's at the pre-emption queue and looks at whether it is
> non-finishable.
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java#L420]
> In the above case, all tasks are speculative but state change is not
> triggering pre-emption queue re-ordering so peek() always returns canFinish
> task even though non-finishable tasks are in the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23234) Optimize TxnHandler::allocateTableWriteIds
[
https://issues.apache.org/jira/browse/HIVE-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108515#comment-17108515
]
Hive QA commented on HIVE-23234:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
30s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 538 unchanged - 3 fixed = 539 total (was 541) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 0 fixed = 187 total (was 186) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to updateCount in
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.setHadoopJobId(String,
long) At
CompactionTxnHandler.java:org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.setHadoopJobId(String,
long) At CompactionTxnHandler.java:[line 1122] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22359/dev-support/hive-personality.sh
|
| git revision | master / a2b3b36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22359/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22359/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22359/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize TxnHandler::allocateTableWriteIds
> --
>
> Key: HIVE-23234
> URL: https://issues.apache.org/jira/browse/HIVE-23234
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments:
[jira] [Resolved] (HIVE-23448) Remove hive-site.xml from input/output/processor payload
[ https://issues.apache.org/jira/browse/HIVE-23448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mustafa Iman resolved HIVE-23448. - Resolution: Won't Fix > Remove hive-site.xml from input/output/processor payload > > > Key: HIVE-23448 > URL: https://issues.apache.org/jira/browse/HIVE-23448 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Mustafa Iman >Assignee: Mustafa Iman >Priority: Major > Attachments: HIVE-23448.patch > > > Depends on https://jira.apache.org/jira/browse/TEZ-4137?filter=-1 > We remove most xml configs from payloads in > https://jira.apache.org/jira/browse/HIVE-23175 > However, hive-site.xml could not be removed from those configs in early stage > for reasons outlined in that jira. > This Jira removes hive-site.xml configs from configuration just before > serializing payloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108475#comment-17108475
]
Hive QA commented on HIVE-23347:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13003031/HIVE-23347.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 17276 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[msck_repair_5]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[msck_repair_6]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[msck_repair_drop]
(batchId=5)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_materialized_view_rewrite_ssb]
(batchId=129)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryEnabledBothTableTypes
(batchId=153)
org.apache.hadoop.hive.ql.exec.TestMsckCreatePartitionsInBatches.testBatchingWhenException
(batchId=291)
org.apache.hadoop.hive.ql.exec.TestMsckCreatePartitionsInBatches.testEqualNumberOfPartitions
(batchId=291)
org.apache.hadoop.hive.ql.exec.TestMsckCreatePartitionsInBatches.testNumberOfCreatePartitionCalls
(batchId=291)
org.apache.hadoop.hive.ql.exec.TestMsckCreatePartitionsInBatches.testSmallNumberOfPartitions
(batchId=291)
org.apache.hadoop.hive.ql.exec.TestMsckCreatePartitionsInBatches.testUnevenNumberOfCreatePartitionCalls
(batchId=291)
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
(batchId=269)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22358/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22358/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22358/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13003031 - PreCommit-HIVE-Build
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Adesh Kumar Rao
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch,
> HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For the following scenario, we expect MSCK REPAIR to discover partitions but
> it couldn't.
> 1. Have partitioned data path as follows.
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10
> hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11
> 2. create external table t1 (key int, value string) partitioned by (Year int,
> Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1'';
> 3. msck repair table t1;
> 4. show partitions t1; --> Returns zero partitions
> 5. select * from t1; --> Returns empty data.
> When the partition directory names are changed to lower case, this works fine.
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=10
> hdfs://mycluster/datapath/t1/year=2020/month=03/day=11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[
https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108468#comment-17108468
]
Hive QA commented on HIVE-23347:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
9s{color} | {color:blue} standalone-metastore/metastore-common in master has 35
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
28s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
16s{color} | {color:red} standalone-metastore/metastore-common generated 1 new
+ 34 unchanged - 1 fixed = 35 total (was 35) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| | org.apache.hadoop.hive.metastore.CheckResult$PartitionResult defines
equals and uses Object.hashCode() At CheckResult.java:Object.hashCode() At
CheckResult.java:[lines 175-180] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22358/dev-support/hive-personality.sh
|
| git revision | master / a2b3b36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22358/yetus/diff-checkstyle-standalone-metastore_metastore-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22358/yetus/new-findbugs-standalone-metastore_metastore-common.html
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22358/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MSCK REPAIR cannot discover partitions with upper case directory names.
> ---
>
> Key: HIVE-23347
> URL: https://issues.apache.org/jira/browse/HIVE-23347
>
[jira] [Commented] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[
https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108423#comment-17108423
]
Hive QA commented on HIVE-23292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002999/HIVE-23292.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 180 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=230)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=230)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=230)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=236)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=236)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binary_output_format]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_1]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_1]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_2]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_3]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_4]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_5]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_6]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_7]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_8]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin11]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin8]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative3]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_map_ppr]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_8]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[regexp_extract]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[serde_user_properties]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sort_merge_join_desc_5]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sort_merge_join_desc_6]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sort_merge_join_desc_7]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_partition_pruning]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[transform_ppr1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[transform_ppr2]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_column_buckets]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_column_list_bucket]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_explode] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_explode]
(batchId=10)
[jira] [Commented] (HIVE-23292) Reduce PartitionDesc payload in MapWork
[
https://issues.apache.org/jira/browse/HIVE-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108404#comment-17108404
]
Hive QA commented on HIVE-23292:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
41s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22357/dev-support/hive-personality.sh
|
| git revision | master / a2b3b36 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22357/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Reduce PartitionDesc payload in MapWork
> ---
>
> Key: HIVE-23292
> URL: https://issues.apache.org/jira/browse/HIVE-23292
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23292.1.patch
>
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java#L105
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anishek Agarwal updated HIVE-23432: --- Resolution: Fixed Status: Resolved (was: Patch Available) merged to master. Thanks for the patch [~aasha] and review [~pkumarsinha] > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch, > HIVE-23432.06.patch, HIVE-23432.07.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[
https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108371#comment-17108371
]
Hive QA commented on HIVE-23281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002998/HIVE-23281.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 226 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[msck_repair_acid]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=15)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_static]
(batchId=122)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=30)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_direct_insert_insert_overwrite]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[alter_table_location2]
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=33)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_dp]
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_insert_overwrite]
(batchId=81)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_mapjoin]
(batchId=45)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=100)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_nullscan]
(batchId=98)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats2]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats3]
(batchId=48)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats4]
(batchId=93)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_stats]
(batchId=72)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_subquery]
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_table_directories_test]
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_table_stats]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_partition]
(batchId=104)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_4]
(batchId=46)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers_acid2]
(batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[change_allowincompatible_vectorization_false_date2]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[change_allowincompatible_vectorization_false_date3]
(batchId=71)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[change_allowincompatible_vectorization_false_date]
(batchId=105)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=52)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[create_transactional_full_acid]
(batchId=107)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[create_transactional_insert_only]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=110)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=72)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_all_partitioned]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_where_partitioned]
(batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_whole_partition]
(batchId=44)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_bucketing]
(batchId=117)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=82)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=52)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explain_locks]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_default_keyword]
(batchId=37)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_only_empty_query]
(batchId=107)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_overwrite]
(batchId=53)
[jira] [Commented] (HIVE-23281) ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to DB for ACID tables
[
https://issues.apache.org/jira/browse/HIVE-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108328#comment-17108328
]
Hive QA commented on HIVE-23281:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
34s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 586 unchanged - 1 fixed = 588 total (was 587) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
45s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 0 fixed = 187 total (was 186) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Dead store to parameters in
org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(String,
String, String, String) At
ObjectStore.java:org.apache.hadoop.hive.metastore.ObjectStore.getNumPartitionsByFilter(String,
String, String, String) At ObjectStore.java:[line 4010] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22356/dev-support/hive-personality.sh
|
| git revision | master / 47d2fe1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22356/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22356/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22356/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ObjectStore::convertToStorageDescriptor can be optimised to reduce calls to
> DB for ACID tables
> --
>
> Key: HIVE-23281
> URL: https://issues.apache.org/jira/browse/HIVE-23281
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23281.1.patch, image-2020-04-23-13-56-17-210.png
>
>
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1980]
>
>
[jira] [Assigned] (HIVE-23475) Track MJ HashTable mem usage
[ https://issues.apache.org/jira/browse/HIVE-23475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis reassigned HIVE-23475: - > Track MJ HashTable mem usage > > > Key: HIVE-23475 > URL: https://issues.apache.org/jira/browse/HIVE-23475 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[ https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23440: -- Attachment: HIVE-23440.04.patch > Move q tests to TestMiniLlapLocal from TestCliDriver where the output is > different, batch 4 > --- > > Key: HIVE-23440 > URL: https://issues.apache.org/jira/browse/HIVE-23440 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch, > HIVE-23440.03.patch, HIVE-23440.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108313#comment-17108313 ] Zoltan Chovan commented on HIVE-23391: -- [~dkuzmenko] could you review this? > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23391) Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED
[ https://issues.apache.org/jira/browse/HIVE-23391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23391: - Attachment: HIVE-23391.3.patch > Change requested lock for ALTER TABLE ADD COLUMN to DDL_SHARED > -- > > Key: HIVE-23391 > URL: https://issues.apache.org/jira/browse/HIVE-23391 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23391.2.patch, HIVE-23391.3.patch, HIVE-23391.patch > > > A long running query can block a simple add column query, as the add column > will require a DDL_EXCLUSIVE lock currently. By changing this to a shared > lock, this metadata only query can be executed without having to wait for the > previous query to finish. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-18882) Minor Logging Improvements in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-18882:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master! Thanks!
> Minor Logging Improvements in Hive Metastore Client Connection
> --
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: noob
> Fix For: 4.0.0
>
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch,
> HIVE-18882.3.patch, HIVE-18882.4.patch, HIVE-18882.5.patch,
> HIVE-18882.5.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if (LOG.isDebugEnabled()) {
> LOG.warn("Failed to connect to the MetaStore Server...", e);
> } else {
> // Don't print full exception trace if DEBUG is not on.
> LOG.warn("Failed to connect to the MetaStore Server...");
> }
> {code}
> I do not understand the logic here. I always want to see the reason for the
> failure. Otherwise, I do not know why it is failing unless I restart the
> server with debug logging enabled. By that point, the error may have
> cleared. Please just use the Exception in the WARN output without adding
> confusing logging for debugging. This is never an expected behavior... that
> enabling debug would change a _warn_ level log message.
> Also... please remove the ellipsis, they add no value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23434?focusedWorklogId=433697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433697
]
ASF GitHub Bot logged work on HIVE-23434:
-
Author: ASF GitHub Bot
Created on: 15/May/20 13:47
Start Date: 15/May/20 13:47
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1017:
URL: https://github.com/apache/hive/pull/1017#discussion_r425813259
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRule.java
##
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.RelFactories.ProjectFactory;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableBitSet.Builder;
+import org.apache.hadoop.hive.ql.exec.DataSketchesFunctions;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate;
+import org.apache.hive.plugin.api.HiveUDFPlugin.UDFDescriptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+/**
+ * This rule could rewrite aggregate calls to be calculated using sketch based
functions.
+ *
+ *
+ * Currently it can rewrite:
+ *
+ * {@code count(distinct(x))} to distinct counting sketches
+ * {@code percentile_cont(0.2) within group (order by id)}
+ *
+ *
+ *
+ * The transformation here works on Aggregate nodes; the operations done are
the following:
+ *
+ *
+ * Identify candidate aggregate calls
+ * A new Project is inserted below the Aggregate; to help with data
pre-processing
+ * A new Aggregate is created in which the aggregation is done by the
sketch function
+ * A new Project is inserted on top of the Aggregate; which unwraps the
resulting
+ *count-distinct estimation from the sketch representation
+ *
+ */
+public final class HiveRewriteToDataSketchesRule extends RelOptRule {
+
+ protected static final Logger LOG =
LoggerFactory.getLogger(HiveRewriteToDataSketchesRule.class);
+ private final Optional countDistinctSketchType;
+ private final Optional percentileContSketchType;
+ private final ProjectFactory projectFactory;
+
+ public HiveRewriteToDataSketchesRule(Optional
countDistinctSketchType,
+ Optional percentileContSketchType) {
+super(operand(HiveAggregate.class, any()));
+this.countDistinctSketchType = countDistinctSketchType;
+this.percentileContSketchType = percentileContSketchType;
+projectFactory = HiveRelFactories.HIVE_PROJECT_FACTORY;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final Aggregate aggregate = call.rel(0);
+
+if (aggregate.getGroupSets().size() != 1) {
+ // not yet supported
+ return;
+}
+
+List newAggCalls = new ArrayList();
+
+VBuilder vb = new VBuilder(aggregate);
+
+if (aggregate.getAggCallList().equals(vb.newAggCalls)) {
+ // rule didn't made any changes
+ return;
+}
+
+newAggCalls = vb.newAggCalls;
+List filedNames=new ArrayList();
+for (int i=0;i newAggCalls;
Review comment:
actually I
[jira] [Commented] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[
https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108300#comment-17108300
]
Hive QA commented on HIVE-23440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002994/HIVE-23440.03.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[udtf_explode]
(batchId=81)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22355/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22355/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22355/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002994 - PreCommit-HIVE-Build
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is
> different, batch 4
> ---
>
> Key: HIVE-23440
> URL: https://issues.apache.org/jira/browse/HIVE-23440
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch,
> HIVE-23440.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23434?focusedWorklogId=433692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433692
]
ASF GitHub Bot logged work on HIVE-23434:
-
Author: ASF GitHub Bot
Created on: 15/May/20 13:36
Start Date: 15/May/20 13:36
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1017:
URL: https://github.com/apache/hive/pull/1017#discussion_r425806233
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRule.java
##
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.RelFactories.ProjectFactory;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableBitSet.Builder;
+import org.apache.hadoop.hive.ql.exec.DataSketchesFunctions;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate;
+import org.apache.hive.plugin.api.HiveUDFPlugin.UDFDescriptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+/**
+ * This rule could rewrite aggregate calls to be calculated using sketch based
functions.
+ *
+ *
+ * Currently it can rewrite:
+ *
+ * {@code count(distinct(x))} to distinct counting sketches
+ * {@code percentile_cont(0.2) within group (order by id)}
+ *
+ *
+ *
+ * The transformation here works on Aggregate nodes; the operations done are
the following:
+ *
+ *
+ * Identify candidate aggregate calls
+ * A new Project is inserted below the Aggregate; to help with data
pre-processing
+ * A new Aggregate is created in which the aggregation is done by the
sketch function
+ * A new Project is inserted on top of the Aggregate; which unwraps the
resulting
+ *count-distinct estimation from the sketch representation
+ *
+ */
+public final class HiveRewriteToDataSketchesRule extends RelOptRule {
+
+ protected static final Logger LOG =
LoggerFactory.getLogger(HiveRewriteToDataSketchesRule.class);
+ private final Optional countDistinctSketchType;
+ private final Optional percentileContSketchType;
+ private final ProjectFactory projectFactory;
+
+ public HiveRewriteToDataSketchesRule(Optional
countDistinctSketchType,
+ Optional percentileContSketchType) {
+super(operand(HiveAggregate.class, any()));
+this.countDistinctSketchType = countDistinctSketchType;
+this.percentileContSketchType = percentileContSketchType;
+projectFactory = HiveRelFactories.HIVE_PROJECT_FACTORY;
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final Aggregate aggregate = call.rel(0);
+
+if (aggregate.getGroupSets().size() != 1) {
+ // not yet supported
+ return;
+}
+
+List newAggCalls = new ArrayList();
+
+VBuilder vb = new VBuilder(aggregate);
+
+if (aggregate.getAggCallList().equals(vb.newAggCalls)) {
+ // rule didn't made any changes
+ return;
+}
+
+newAggCalls = vb.newAggCalls;
+List filedNames=new ArrayList();
Review comment:
I didn't even noticed this :)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.6.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, HIVE-23354.6.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23434?focusedWorklogId=433691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433691
]
ASF GitHub Bot logged work on HIVE-23434:
-
Author: ASF GitHub Bot
Created on: 15/May/20 13:34
Start Date: 15/May/20 13:34
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1017:
URL: https://github.com/apache/hive/pull/1017#discussion_r425805140
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRewriteToDataSketchesRule.java
##
@@ -0,0 +1,327 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.hep.HepRelVertex;
+import org.apache.calcite.rel.RelCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.RelFactories.ProjectFactory;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.ImmutableBitSet.Builder;
+import org.apache.hadoop.hive.ql.exec.DataSketchesFunctions;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate;
+import org.apache.hive.plugin.api.HiveUDFPlugin.UDFDescriptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+/**
+ * This rule could rewrite aggregate calls to be calculated using sketch based
functions.
+ *
+ *
+ * Currently it can rewrite:
+ *
+ * {@code count(distinct(x))} to distinct counting sketches
Review comment:
great idea! I've added it!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 433691)
Time Spent: 0.5h (was: 20m)
> Add option to rewrite PERCENTILE_CONT to sketch functions
> -
>
> Key: HIVE-23434
> URL: https://issues.apache.org/jira/browse/HIVE-23434
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23434.01.patch, HIVE-23434.02.patch,
> HIVE-23434.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-18882) Minor Logging Improvements in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-18882:
--
Summary: Minor Logging Improvements in Hive Metastore Client Connection
(was: Do Not Hide Exception in Hive Metastore Client Connection)
> Minor Logging Improvements in Hive Metastore Client Connection
> --
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch,
> HIVE-18882.3.patch, HIVE-18882.4.patch, HIVE-18882.5.patch,
> HIVE-18882.5.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if (LOG.isDebugEnabled()) {
> LOG.warn("Failed to connect to the MetaStore Server...", e);
> } else {
> // Don't print full exception trace if DEBUG is not on.
> LOG.warn("Failed to connect to the MetaStore Server...");
> }
> {code}
> I do not understand the logic here. I always want to see the reason for the
> failure. Otherwise, I do not know why it is failing unless I restart the
> server with debug logging enabled. By that point, the error may have
> cleared. Please just use the Exception in the WARN output without adding
> confusing logging for debugging. This is never an expected behavior... that
> enabling debug would change a _warn_ level log message.
> Also... please remove the ellipsis, they add no value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22942) Replace PTest with an alternative
[ https://issues.apache.org/jira/browse/HIVE-22942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22942: Description: I never opened a jira about this...but it might actually help collect ideas and actually start going somewhere sooner than later :D Right now we maintain the ptest2 project inside Hive to be able to run Hive tests in a distributed fashion...the backstab of this solution is that we are putting much effort into maintaining a distributed test execution framework... I think it would be better if we could find an off the shelf solution for the task and migrate to that instead of putting more efforts into the ptest framework some info/etc about how it compares to existing one: https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n was: I never opened a jira about this...but it might actually help collect ideas and actually start going somewhere sooner than later :D Right now we maintain the ptest2 project inside Hive to be able to run Hive tests in a distributed fashion...the backstab of this solution is that we are putting much effort into maintaining a distributed test execution framework... I think it would be better if we could find an off the shelf solution for the task and migrate to that instead of putting more efforts into the ptest framework > Replace PTest with an alternative > - > > Key: HIVE-22942 > URL: https://issues.apache.org/jira/browse/HIVE-22942 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > I never opened a jira about this...but it might actually help collect ideas > and actually start going somewhere sooner than later :D > Right now we maintain the ptest2 project inside Hive to be able to run Hive > tests in a distributed fashion...the backstab of this solution is that we are > putting much effort into maintaining a distributed test execution framework... > I think it would be better if we could find an off the shelf solution for the > task and migrate to that instead of putting more efforts into the ptest > framework > some info/etc about how it compares to existing one: > https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit#heading=h.e51vlxui3e6n -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23473) Handle NPE when ObjectCache is null while getting DynamicValue during ORC split generation
[
https://issues.apache.org/jira/browse/HIVE-23473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108273#comment-17108273
]
Ganesha Shreedhara commented on HIVE-23473:
---
[~jdere], [~ashutoshc] Please review the patch.
> Handle NPE when ObjectCache is null while getting DynamicValue during ORC
> split generation
> --
>
> Key: HIVE-23473
> URL: https://issues.apache.org/jira/browse/HIVE-23473
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-23473.patch
>
>
> NullPointerException is thrown in the following flow.
>
> {code:java}
> java.lang.RuntimeException: ORC split generation failed with exception:
> java.lang.NullPointerException
> Caused by: java.lang.NullPointerException
> at
> org.apache.orc.impl.RecordReaderImpl.compareToRange(RecordReaderImpl.java:312)
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:559)
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463)
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2214)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2190)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2182)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:186)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1477)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1265)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1446)
> .
> .
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1809)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1895)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:526)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:649)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:206)
> {code}
>
> Shouldn't we just throw NoDynamicValuesException when [ObjectCache|#L119]] is
> null instead of returning it similar to how we handled when [conf |#L110]]or
> [DynamicValueRegistry|#L125]] is null while getting dynamic value?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108264#comment-17108264
]
Miklos Gergely commented on HIVE-23460:
---
+1 (pending tests)
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch,
> HIVE-23460.02.patch, HIVE-23460.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23474) Deny Repl Dump if the database is a target of replication
[ https://issues.apache.org/jira/browse/HIVE-23474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi reassigned HIVE-23474: -- > Deny Repl Dump if the database is a target of replication > - > > Key: HIVE-23474 > URL: https://issues.apache.org/jira/browse/HIVE-23474 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23440) Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 4
[
https://issues.apache.org/jira/browse/HIVE-23440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108255#comment-17108255
]
Hive QA commented on HIVE-23440:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22355/dev-support/hive-personality.sh
|
| git revision | master / 390ad7d |
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22355/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is
> different, batch 4
> ---
>
> Key: HIVE-23440
> URL: https://issues.apache.org/jira/browse/HIVE-23440
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23440.01.patch, HIVE-23440.02.patch,
> HIVE-23440.03.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23361) Optimising privilege synchroniser
[
https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108238#comment-17108238
]
Hive QA commented on HIVE-23361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002979/hive-23361.4.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17271 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_analyze]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized]
(batchId=90)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=85)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=70)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb_schq]
(batchId=86)
org.apache.hive.service.server.TestInformationSchemaWithPrivilegeZookeeperPlain.test
(batchId=212)
org.apache.hive.service.server.TestInformationSchemaWithPrivilegeZookeeperSSL.test
(batchId=211)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22354/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22354/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22354/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002979 - PreCommit-HIVE-Build
> Optimising privilege synchroniser
> -
>
> Key: HIVE-23361
> URL: https://issues.apache.org/jira/browse/HIVE-23361
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Simhadri G
>Assignee: Simhadri G
>Priority: Minor
> Attachments: hive-23361.1.patch, hive-23361.2.patch,
> hive-23361.3.patch, hive-23361.4.patch, hive-23361.patch, hive-23361.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> This patch store the privilege information as bit string. This is done to
> reduce the size of the tbl_col_privs tables in metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[ https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108235#comment-17108235 ] Adesh Kumar Rao commented on HIVE-21637: [~daijy] the Jira has been idle for quite some time. If you are not working on it, Can I take over? > Synchronized metastore cache > > > Key: HIVE-21637 > URL: https://issues.apache.org/jira/browse/HIVE-21637 > Project: Hive > Issue Type: New Feature >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-21637-1.patch, HIVE-21637.10.patch, > HIVE-21637.11.patch, HIVE-21637.12.patch, HIVE-21637.13.patch, > HIVE-21637.14.patch, HIVE-21637.15.patch, HIVE-21637.16.patch, > HIVE-21637.17.patch, HIVE-21637.18.patch, HIVE-21637.19.patch, > HIVE-21637.19.patch, HIVE-21637.2.patch, HIVE-21637.20.patch, > HIVE-21637.21.patch, HIVE-21637.22.patch, HIVE-21637.23.patch, > HIVE-21637.24.patch, HIVE-21637.25.patch, HIVE-21637.26.patch, > HIVE-21637.27.patch, HIVE-21637.28.patch, HIVE-21637.29.patch, > HIVE-21637.3.patch, HIVE-21637.30.patch, HIVE-21637.31.patch, > HIVE-21637.32.patch, HIVE-21637.33.patch, HIVE-21637.34.patch, > HIVE-21637.35.patch, HIVE-21637.36.patch, HIVE-21637.37.patch, > HIVE-21637.38.patch, HIVE-21637.39.patch, HIVE-21637.4.patch, > HIVE-21637.40.patch, HIVE-21637.41.patch, HIVE-21637.42.patch, > HIVE-21637.43.patch, HIVE-21637.44.patch, HIVE-21637.45.patch, > HIVE-21637.46.patch, HIVE-21637.47.patch, HIVE-21637.48.patch, > HIVE-21637.49.patch, HIVE-21637.5.patch, HIVE-21637.50.patch, > HIVE-21637.51.patch, HIVE-21637.52.patch, HIVE-21637.53.patch, > HIVE-21637.54.patch, HIVE-21637.55.patch, HIVE-21637.56.patch, > HIVE-21637.57.patch, HIVE-21637.58.patch, HIVE-21637.59.patch, > HIVE-21637.6.patch, HIVE-21637.60.patch, HIVE-21637.61.patch, > HIVE-21637.7.patch, HIVE-21637.8.patch, HIVE-21637.9.patch > > > Currently, HMS has a cache implemented by CachedStore. The cache is > asynchronized and in HMS HA setting, we can only get eventual consistency. In > this Jira, we try to make it synchronized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23470) Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver
[ https://issues.apache.org/jira/browse/HIVE-23470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23470: -- Attachment: HIVE-23470.01.patch > Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with > TestMiniLlapLocalCliDriver > - > > Key: HIVE-23470 > URL: https://issues.apache.org/jira/browse/HIVE-23470 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23470.01.patch > > > Some tests are failing with TestMiniLlapLocalCliDriver, but running fine with > TestMiniLlapCliDriver, let's move them there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23470) Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver
[ https://issues.apache.org/jira/browse/HIVE-23470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23470: -- Status: Patch Available (was: Open) > Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with > TestMiniLlapLocalCliDriver > - > > Key: HIVE-23470 > URL: https://issues.apache.org/jira/browse/HIVE-23470 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23470.01.patch > > > Some tests are failing with TestMiniLlapLocalCliDriver, but running fine with > TestMiniLlapCliDriver, let's move them there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Open (was: Patch Available) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Attachment: HIVE-23347.5.patch > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adesh Kumar Rao updated HIVE-23347: --- Status: Patch Available (was: Open) > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch, HIVE-23347.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Summary: Add Deny Policy on Target Database After Ranger Replication to avoid writes (was: Add Deny Policy on Target Database After Ranger Replication to avoid writes and repl dump) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > --- > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes and repl dump
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Attachment: HIVE-23433.02.patch Status: Patch Available (was: In Progress) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > and repl dump > - > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23433.01.patch, HIVE-23433.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23433) Add Deny Policy on Target Database After Ranger Replication to avoid writes and repl dump
[ https://issues.apache.org/jira/browse/HIVE-23433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23433: --- Status: In Progress (was: Patch Available) > Add Deny Policy on Target Database After Ranger Replication to avoid writes > and repl dump > - > > Key: HIVE-23433 > URL: https://issues.apache.org/jira/browse/HIVE-23433 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23433.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?focusedWorklogId=433659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-433659
]
ASF GitHub Bot logged work on HIVE-23460:
-
Author: ASF GitHub Bot
Created on: 15/May/20 12:16
Start Date: 15/May/20 12:16
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1018:
URL: https://github.com/apache/hive/pull/1018#discussion_r425760827
##
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestDisabledHandler.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.qoption;
+
+import org.apache.hadoop.hive.ql.QTestUtil;
+import org.junit.Assume;
+
+import com.google.common.base.Strings;
+
+/**
+ * QTest disabled directive handler
+ *
+ * Example:
+ * --! qt:disabled:reason
+ *
+ */
+public class QTestDisabledHandler implements QTestOptionHandler {
+
+ private String message;
+
+ @Override
+ public void processArguments(String arguments) {
+message = arguments;
+if (Strings.isNullOrEmpty(message)) {
+ throw new RuntimeException("you have to give a reason why it was
ignored");
+}
+ }
+
+ @Override
+ public void beforeTest(QTestUtil qt) throws Exception {
+Assume.assumeTrue(message, (message == null));
Review comment:
the assume exception also transfers the message:
http://34.66.156.144:8080/job/hive-precommit/view/change-requests/job/PR-7/122/testReport/org.apache.hadoop.hive.cli/TestAccumuloCliDriver/Testing___split_11___Archive___testCliDriver_accumulo_joins_/
I don't want to add a boolean in "QTestUtil" - because that way I would
need to also pass the message...which would break to conciseness of this
feature.
I had to catch and ethrow those exceptions to not "fail" these tests;
instead show the reason why they are ignored.
The problem is that there are "catch Excpetion" stuff all over the place -
they could probably placed into finally...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 433659)
Time Spent: 0.5h (was: 20m)
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch,
> HIVE-23460.02.patch, HIVE-23460.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23361) Optimising privilege synchroniser
[
https://issues.apache.org/jira/browse/HIVE-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108222#comment-17108222
]
Hive QA commented on HIVE-23361:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
34s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22354/dev-support/hive-personality.sh
|
| git revision | master / 390ad7d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: metastore ql itests itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22354/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimising privilege synchroniser
> -
>
> Key: HIVE-23361
> URL: https://issues.apache.org/jira/browse/HIVE-23361
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Simhadri G
>Assignee: Simhadri G
>Priority: Minor
> Attachments: hive-23361.1.patch, hive-23361.2.patch,
> hive-23361.3.patch, hive-23361.4.patch, hive-23361.patch, hive-23361.patch
>
>
> Privilege synchronizer pulls the list of databases, tables and columns from
> the Hive Metastore. For each of these objects it fetches the privilege
> information and invokes HMS API to refresh the privilege information in HMS.
> This patch store the privilege information as bit string. This is done to
> reduce the size of the tbl_col_privs tables in metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23374) QueryDisplay must be threadsafe
[
https://issues.apache.org/jira/browse/HIVE-23374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108209#comment-17108209
]
László Bodor commented on HIVE-23374:
-
okay, I'm not aware of the exact usage of this class, but it makes sense to
follow the pattern with getQueryStartTime as well
+1
> QueryDisplay must be threadsafe
> ---
>
> Key: HIVE-23374
> URL: https://issues.apache.org/jira/browse/HIVE-23374
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23374.01.patch, HIVE-23374.02.patch
>
>
> apparently right now it might bump into
> java.util.ConcurrentModificationException -s
> http://34.66.156.144:8080//job/hive-precommit/job/PR-7/41//testReport/junit/org.apache.hive.jdbc/TestJdbcWithMiniLlapRow/Testing___split_23___Archive___testInvalidReferenceCountScenario/
> {code}
> org.apache.hive.service.cli.HiveSQLException
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:360)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:346)
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:352)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:239)
> at
> org.apache.hive.jdbc.BaseJdbcWithMiniLlap.testInvalidReferenceCountScenario(BaseJdbcWithMiniLlap.java:396)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at
> java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at java.util.ArrayList.addAll(ArrayList.java:581)
> at
> org.apache.hadoop.hive.ql.QueryDisplay.getTaskDisplays(QueryDisplay.java:273)
> at
> org.apache.hive.service.cli.operation.SQLOperation.getTaskStatus(SQLOperation.java:494)
> at
> org.apache.hive.service.cli.operation.Operation.getStatus(Operation.java:141)
> at
> org.apache.hive.service.cli.CLIService.getOperationStatus(CLIService.java:478)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetOperationStatus(ThriftCLIService.java:697)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1757)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1742)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108206#comment-17108206
]
László Bodor commented on HIVE-23460:
-
+1, pending tests
this way we won't have the chance to remove qtest from a driver but not from
other afaik, however, it's not important
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch,
> HIVE-23460.02.patch, HIVE-23460.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23314) Upgrade to Kudu 1.12
[ https://issues.apache.org/jira/browse/HIVE-23314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108189#comment-17108189 ] Miklos Gergely commented on HIVE-23314: --- +1 (pending tests) > Upgrade to Kudu 1.12 > > > Key: HIVE-23314 > URL: https://issues.apache.org/jira/browse/HIVE-23314 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23314.01.patch > > > we need KUDU-3044 because it could cause random failures... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[
https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108187#comment-17108187
]
Hive QA commented on HIVE-23354:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002975/HIVE-23354.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=78)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22353/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22353/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22353/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002975 - PreCommit-HIVE-Build
> Remove file size sanity checking from compareTempOrDuplicateFiles
> -
>
> Key: HIVE-23354
> URL: https://issues.apache.org/jira/browse/HIVE-23354
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch,
> HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch
>
>
> [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010]
> compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to
> determine which file(s) to keep.
> I've seen instances where this function throws an exception due to the fact
> that the newer attemptId file size is less than the older attemptId (thus
> failing the query).
> I think this assumption is faulty, due to various factors such as file
> compression and the order in which values are written. It may be prudent to
> trust that the newest attemptId is in fact the best choice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23396) Many fixes and improvements to stabilize tests
[
https://issues.apache.org/jira/browse/HIVE-23396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108171#comment-17108171
]
Miklos Gergely commented on HIVE-23396:
---
+1 (pending tests)
> Many fixes and improvements to stabilize tests
> --
>
> Key: HIVE-23396
> URL: https://issues.apache.org/jira/browse/HIVE-23396
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HHIVE-23396.01.patch, HIVE-23396.01.patch,
> HIVE-23396.02.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> sometimes it seems to fail ; there were also some derby exceptions earlier
> the test itself has a @Before annotated setUp ; which starts a metastore - I
> think this should be done only once ; so @BeforeClass might be a better
> option...
> {code}
> org.apache.hadoop.hive.metastore.api.NoSuchObjectException: database
> hive.default
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.java:719)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_252]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_252]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_252]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy41.getDatabase(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2076)
> ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2324)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_252]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_252]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_252]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy43.create_table_req(Unknown Source) [?:?]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16497)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16481)
> [hive-standalone-metastore-common-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_252]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_252]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> [hadoop-common-3.1.0.jar:?]
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> [hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [libthrift-0.9.3-1.jar:0.9.3-1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_252]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_252]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
> 2020-05-06T17:04:30,243 ERROR [pool-9-thread-3]
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[
https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108170#comment-17108170
]
Hive QA commented on HIVE-23354:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 2 new + 228 unchanged - 2
fixed = 230 total (was 230) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22353/dev-support/hive-personality.sh
|
| git revision | master / 390ad7d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22353/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22353/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove file size sanity checking from compareTempOrDuplicateFiles
> -
>
> Key: HIVE-23354
> URL: https://issues.apache.org/jira/browse/HIVE-23354
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch,
> HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch
>
>
> [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010]
> compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to
> determine which file(s) to keep.
> I've seen instances where this function throws an exception due to the fact
> that the newer attemptId file size is less than the older attemptId (thus
> failing the query).
> I think this assumption is faulty, due to various factors such as file
> compression and the
[jira] [Commented] (HIVE-23347) MSCK REPAIR cannot discover partitions with upper case directory names.
[ https://issues.apache.org/jira/browse/HIVE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108164#comment-17108164 ] Sankar Hariappan commented on HIVE-23347: - [~nareshpr], [~adeshrao] I think, Hive should treat Year=2020/Month=3/Day=2 and year=2020/month=3/day=2 as same partition. If both paths exists in storage, then MSCK should throw error as duplication partition directory. However, dept=Sales and dept=sales are not same partition where dept is String column. Hive just treats the partition column name as case insensitive but partition values as case sensitive. > MSCK REPAIR cannot discover partitions with upper case directory names. > --- > > Key: HIVE-23347 > URL: https://issues.apache.org/jira/browse/HIVE-23347 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.0 >Reporter: Sankar Hariappan >Assignee: Adesh Kumar Rao >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23347.01.patch, HIVE-23347.2.patch, > HIVE-23347.3.patch, HIVE-23347.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > For the following scenario, we expect MSCK REPAIR to discover partitions but > it couldn't. > 1. Have partitioned data path as follows. > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=10 > hdfs://mycluster/datapath/t1/Year=2020/Month=03/Day=11 > 2. create external table t1 (key int, value string) partitioned by (Year int, > Month int, Day int) stored as orc location hdfs://mycluster/datapath/t1''; > 3. msck repair table t1; > 4. show partitions t1; --> Returns zero partitions > 5. select * from t1; --> Returns empty data. > When the partition directory names are changed to lower case, this works fine. > hdfs://mycluster/datapath/t1/year=2020/month=03/day=10 > hdfs://mycluster/datapath/t1/year=2020/month=03/day=11 -- This message was sent by Atlassian Jira (v8.3.4#803005)
