[jira] [Created] (DRILL-6236) batch sizing for hash join and hash aggregate
Padma Penumarthy created DRILL-6236: --- Summary: batch sizing for hash join and hash aggregate Key: DRILL-6236 URL: https://issues.apache.org/jira/browse/DRILL-6236 Project: Apache Drill Issue Type: Improvement Components: Execution - Flow Affects Versions: 1.13.0 Reporter: Padma Penumarthy Assignee: Padma Penumarthy Fix For: 1.14.0 limit output batch size for hash join and hash aggregate based on memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6234) VarCharVector setValueCount can throw IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396335#comment-16396335
]
Paul Rogers commented on DRILL-6234:
[~timothyfarkas], the tricky bit here is that your proposal is not simply a bug
fix; it is a change of design. Now, the way that the value count works is
decidedly odd: one would naively think that each vector would know how much
data it holds. The value count name itself is misleading; all that call really
does is set the end-of-buffer location for the underlying {{ByteBuf}}. (This
is, in turn, because a value vector is not like a database buffer, it is
instead modeled on a network serialization/deserialization buffer.)
The call in {{VarCharVector}} fails because we need to set two positions: the
end of the offsets and the end of the values. To get the end of the values, we
index into the offsets to find the position of the (non-existent) n+1st item,
and use that as the length of the values.
If we've not written any values, than the offsets are all zero (if the vector
has been resized, or if this is the first allocation from direct memory) or
filled with garbage (if this is the initial vector allocation and the buffer
has been reused from the free list in which case we don't zero the vector.)
So, it turns out, *there is no way* to set the end position of the values
vector until we write data. This is obviously not true of a fixed vector (such
as the offsets) since we know the width of each value. But, it is very true of
variable-width vectors (VarChar) or arrays.
Bottom line: find some other way to accomplish your goal. The Drill protocol
expects the client to keep track of the item count until the vector is
finished. For example, that is one of the tasks that the result set loader
handles: keeping track of the top-level values, and the counts of values in
nested arrays of maps of arrays of...
> VarCharVector setValueCount can throw IndexOutOfBoundsException
> ---
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized you should not call get for that index.
> {code}
> at
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
> at
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114)
> at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484)
> at
> org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432)
> at
> org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount(VarCharVector.java:729)
> at
> org.apache.drill.exec.vector.VarCharVectorTest.testExpandingNonEmptyVectorSetValueCount(VarCharVectorTest.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396256#comment-16396256
]
ASF GitHub Bot commented on DRILL-6005:
---
Github user dvjyothsna commented on the issue:
https://github.com/apache/drill/pull/1162
@arina-ielchiieva Please review..Thank you
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396255#comment-16396255
]
ASF GitHub Bot commented on DRILL-6005:
---
GitHub user dvjyothsna opened a pull request:
https://github.com/apache/drill/pull/1162
DRILL-6005: Fixed TestGracefulShutdown tests to skip check for loopba…
…ck address usage in distributed mode
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dvjyothsna/drill DRILL-6005
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1162.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1162
commit 38cf91bfd3b24ed458d8519129001632ad940048
Author: dvjyothsna
Date: 2018-03-12T22:38:13Z
DRILL-6005: Fixed TestGracefulShutdown tests to skip check for loopback
address usage in distributed mode
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6234) VarCharVector setValueCount can throw IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6234:
--
Description:
Doing the following will throw an Index out of bounds exception.
{code}
final VarCharVector vector = new VarCharVector(field, allocator);
vector.allocateNew();
vector.getMutator().setValueCount(100);
{code}
The expected behavior is to resize the array appropriately. If an index is
uninitialized you should not call get for that index.
{code}
at
org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
at
org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114)
at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484)
at
org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432)
at
org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount(VarCharVector.java:729)
at
org.apache.drill.exec.vector.VarCharVectorTest.testExpandingNonEmptyVectorSetValueCount(VarCharVectorTest.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
was:
Doing the following will throw an Index out of bounds exception.
{code}
final VarCharVector vector = new VarCharVector(field, allocator);
vector.allocateNew();
vector.getMutator().setValueCount(100);
{code}
The expected behavior is to resize the array appropriately. If an index is
uninitialized you should not call get for that index.
> VarCharVector setValueCount can throw IndexOutOfBoundsException
> ---
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized you should not call get for that index.
> {code}
> at
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
> at
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114)
> at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484)
> at
> org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432)
> at
> org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount(VarCharVector.java:729)
> at
> org.apache.drill.exec.vector.VarCharVectorTest.testExpandingNonEmptyVectorSetValueCount(VarCharVectorTest.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6234) VarCharVector setValueCount can throw IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396243#comment-16396243
]
Timothy Farkas commented on DRILL-6234:
---
[~paul-rogers] The motivation for this was to make handling cases where you
don't know the amount of data that will go into the vector upfront a little
simpler. Such a case is a BatchHolder in a hash table where it is desirable to
grow the vectors as you get more data instead preallocating a large chunk of
memory that could go unused. You are right, an index out of bound exception
does occur when setting more elements than there are in the vector, however, I
assumed things would behave as they do with the FixedWidth vectors. For example
Uint4Vector resizes the array to fit valueCount elements when setValueCount is
called even if setValueCount. So if we can make the behavior consistent without
any negative side effects, then why not do it?
> VarCharVector setValueCount can throw IndexOutOfBoundsException
> ---
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized you should not call get for that index.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6234) VarCharVector setValueCount can throw IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6234:
--
Summary: VarCharVector setValueCount can throw IndexOutOfBoundsException
(was: VarCharVector setValueCount can throw out of memory exception)
> VarCharVector setValueCount can throw IndexOutOfBoundsException
> ---
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized it can return an empty byte array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6234) VarCharVector setValueCount can throw IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6234:
--
Description:
Doing the following will throw an Index out of bounds exception.
{code}
final VarCharVector vector = new VarCharVector(field, allocator);
vector.allocateNew();
vector.getMutator().setValueCount(100);
{code}
The expected behavior is to resize the array appropriately. If an index is
uninitialized you should not call get for that index.
was:
Doing the following will throw an Index out of bounds exception.
{code}
final VarCharVector vector = new VarCharVector(field, allocator);
vector.allocateNew();
vector.getMutator().setValueCount(100);
{code}
The expected behavior is to resize the array appropriately. If an index is
uninitialized it can return an empty byte array.
> VarCharVector setValueCount can throw IndexOutOfBoundsException
> ---
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized you should not call get for that index.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6234) VarCharVector setValueCount can throw out of memory exception
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396148#comment-16396148
]
Paul Rogers commented on DRILL-6234:
As it turns out, {{setValueCount()}} is descriptive, not proscriptive. That is,
{{setValueCount()}} says how many items are actually in the vector, not how
many you would like to hold. Use {{allocateNew()}} to specify the number of
entries (and entry width) you'd like to hold.
In general, the pattern that Drill use (for better or worse) is:
* Create the vector instance
* Allocate the vector memory with {{allocateNew()}}
* Write values into the vector.
* Report the number of items written with {{setValueCount()}}.
If you get an error, it is likely because you are trying to set the value count
to greater than the actual number of entries.
There was a known bug that setting the count to 0, when the vector was
unallocated, lead to an error.
> VarCharVector setValueCount can throw out of memory exception
> -
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized it can return an empty byte array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6223) Drill fails on Schema changes
[ https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6223: - Summary: Drill fails on Schema changes (was: Drill fails on Schema hanges ) > Drill fails on Schema changes > -- > > Key: DRILL-6223 > URL: https://issues.apache.org/jira/browse/DRILL-6223 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.10.0, 1.12.0 >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > Drill Query Failing when selecting all columns from a Complex Nested Data > File (Parquet) Set). There are differences in Schema among the files: > * The Parquet files exhibit differences both at the first level and within > nested data types > * A select * will not cause an exception but using a limit clause will > * Note also this issue seems to happen only when multiple Drillbit minor > fragments are involved (concurrency higher than one) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6235) Flatten query leads to out of memory in RPC layer.
Khurram Faraaz created DRILL-6235:
-
Summary: Flatten query leads to out of memory in RPC layer.
Key: DRILL-6235
URL: https://issues.apache.org/jira/browse/DRILL-6235
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.12.0
Reporter: Khurram Faraaz
Assignee: Padma Penumarthy
Attachments: 25593391-512d-23ab-7c84-3651006931e2.sys.drill
Flatten query leads to out of memory in RPC layer. Query profile is attached
here.
Total number of JSON files = 4095
Each JSON file has nine rows
And each row in the JSON has an array with 1024 integer values, and there are
other string values outside of the array.
Two major fragments and eighty eight minor fragments were created
On a 4 node CentOS cluster
number of CPU cores
[root@qa102-45 ~]# grep -c ^processor /proc/cpuinfo
32
Details of memory
{noformat}
0: jdbc:drill:schema=dfs.tmp> select * from sys.memory;
+--++---+-+-+-+-+
| hostname | user_port | heap_current | heap_max | direct_current |
jvm_direct_current | direct_max |
+--++---+-+-+-+-+
| qa102-45.qa.lab | 31010 | 1130364912 | 4294967296 | 0 | 170528 | 8589934592 |
| qa102-47.qa.lab | 31010 | 171823104 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-48.qa.lab | 31010 | 201326576 | 4294967296 | 0 | 21912 | 8589934592 |
| qa102-46.qa.lab | 31010 | 214780896 | 4294967296 | 0 | 21912 | 8589934592 |
+--++---+-+-+-+-+
4 rows selected (0.166 seconds)
{noformat}
Reset all options and set slice_target=1
alter system reset all;
alter system set `planner.slice_target`=1;
{noformat}
SELECT * , FLATTEN(arr) FROM many_json_files
...
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the
query.
Failure allocating buffer.
Fragment 1:38
[Error Id: cf4fd273-d8a2-45e8-8d72-15c738e53b0f on qa102-45.qa.lab:31010]
(state=,code=0)
{noformat}
Stack trace from drillbit.log fir above failing query.
{noformat}
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 25593391-512d-23ab-7c84-3651006931e2:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
2018-03-12 11:52:33,849 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 25593391-512d-23ab-7c84-3651006931e2:0:0:
State to report: RUNNING
2018-03-12 11:52:33,854 [25593391-512d-23ab-7c84-3651006931e2:frag:0:0] INFO
o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option:
true
2018-03-12 11:52:35,929 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 92340224.
2018-03-12 11:52:35,929 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 92340224.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,930 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,930 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-3] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 83886080.
2018-03-12 11:52:35,931 [BitServer-3] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,931 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
...
...
2018-03-12 11:52:35,939 [BitServer-4] WARN o.a.d.exec.rpc.ProtobufLengthDecoder
- Failure allocating buffer on incoming stream due to memory limits. Current
Allocation: 67174400.
2018-03-12 11:52:35,939 [BitServer-4] ERROR o.a.drill.exec.rpc.data.DataServer
- Out of memory in RPC layer.
2018-03-12 11:52:35,939 [BitServer-2] WARN o.a.d.exec.rpc.Pr
[jira] [Commented] (DRILL-6234) VarCharVector setValueCount can throw out of memory exception
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395737#comment-16395737
]
Vlad Rozov commented on DRILL-6234:
---
[~timothyfarkas] please provide stack trace.
> VarCharVector setValueCount can throw out of memory exception
> -
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized it can return an empty byte array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4120) dir0 does not work when the directory structure contains Avro files
[ https://issues.apache.org/jira/browse/DRILL-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395718#comment-16395718 ] Boaz Ben-Zvi commented on DRILL-4120: - Merged with commit ID - 4652b0ba4f9a0708227e2b83a7097ff0517df33e > dir0 does not work when the directory structure contains Avro files > --- > > Key: DRILL-4120 > URL: https://issues.apache.org/jira/browse/DRILL-4120 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization, Storage - Avro >Affects Versions: 1.3.0 >Reporter: Stefán Baxter >Assignee: Volodymyr Vysotskyi >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > Attachments: > 0001-DRILL-4120-Support-reading-directories-having-avro-f.patch > > > Any select statment containing dirN fails if the target directory structure > contains Avro files. > Steps to test: > 1. create a simple directory structure > 2. copy an avro file into each directory > 3. execute a query containing dir0 > outcome: > Error: VALIDATION ERROR: From line 1, column 117 to line 1, column 120: > Column 'dir0' not found in any table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6218) Update release profile to not generate MD5 checksum
[ https://issues.apache.org/jira/browse/DRILL-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395721#comment-16395721 ] Boaz Ben-Zvi commented on DRILL-6218: - Merged with commit ID - dfe47ce04cae8ef9af08d67c91f2bbbdfacf09ed > Update release profile to not generate MD5 checksum > --- > > Key: DRILL-6218 > URL: https://issues.apache.org/jira/browse/DRILL-6218 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.12.0 >Reporter: Parth Chandra >Assignee: Parth Chandra >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > > Latest release guidelines require that we not generate a MD5 checksum. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6021) Show shutdown button when authentication is not enabled
[
https://issues.apache.org/jira/browse/DRILL-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395715#comment-16395715
]
Boaz Ben-Zvi commented on DRILL-6021:
-
Merged with commit ID - e61df7c8d0da192c5c2c9e64b1d213c63cc81ecb
> Show shutdown button when authentication is not enabled
> ---
>
> Key: DRILL-6021
> URL: https://issues.apache.org/jira/browse/DRILL-6021
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> After DRILL-6017 {{shouldShowAdminInfo}} is used to decide if shutdown button
> should be displayed on index page. But this option is set to true when
> authentication is enabled and user is an admin. When authentication is not
> enabled, user by default is admin. So with this fix without authentication,
> shutdown button is absent but should be present.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh
[ https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395713#comment-16395713 ] Boaz Ben-Zvi commented on DRILL-6040: - Merged with commit ID - c5ae560dcc3fba6d1949be17d07a22085e4d49fb > Need to add usage for graceful_stop to drillbit.sh > -- > > Key: DRILL-6040 > URL: https://issues.apache.org/jira/browse/DRILL-6040 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.13.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > > git.commit.id.abbrev=eb0c403 > Usage for graceful_stop is missing from drillbit.sh. > ./drillbit.sh > Usage: drillbit.sh [--config|--site ] > (start|stop|status|restart|run) [args] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6151) Fragment executors may terminate without sending final batch to a downstream causing query to hang
[ https://issues.apache.org/jira/browse/DRILL-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395709#comment-16395709 ] Boaz Ben-Zvi commented on DRILL-6151: - Merged with commit ID - fe05ba001986f9c8fc3cc34387353383e93b433a > Fragment executors may terminate without sending final batch to a downstream > causing query to hang > -- > > Key: DRILL-6151 > URL: https://issues.apache.org/jira/browse/DRILL-6151 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6203) Repeated Map Vector does not give correct payload bytecount.
[ https://issues.apache.org/jira/browse/DRILL-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boaz Ben-Zvi resolved DRILL-6203. - Resolution: Fixed Merged with commit ID - 0a8a3f1984b17c883dd8ee7cc1065932a4700224 > Repeated Map Vector does not give correct payload bytecount. > - > > Key: DRILL-6203 > URL: https://issues.apache.org/jira/browse/DRILL-6203 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > Repeated Map Vector does not give correct payload byte count. It calls > abstractMapVector method which gives payload byte count for a given value > count for simple map (non repetitive) case. We need to overload this method > for repeated map to get the right numbers. > Also, Repeated Map and Repeated List vectors implement > RepeatedFixedWidthVectorLike and that causes problems when doing memory > allocation with AllocationHelper routines for those vectors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6162) Enhance record batch sizer to retain nesting information for map columns.
[ https://issues.apache.org/jira/browse/DRILL-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boaz Ben-Zvi resolved DRILL-6162. - Resolution: Fixed Merged with commit ID - 47c5d1feaaaf9b6384ed8ef1011fa58b9272b362 > Enhance record batch sizer to retain nesting information for map columns. > - > > Key: DRILL-6162 > URL: https://issues.apache.org/jira/browse/DRILL-6162 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > Enhance the record batch sizer to maintain the columnSizes in nested fashion > for maps so given a column, we can get sizing information of all children > underneath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6126) Allocate memory for value vectors upfront in flatten operator
[ https://issues.apache.org/jira/browse/DRILL-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395700#comment-16395700 ] Boaz Ben-Zvi commented on DRILL-6126: - Merged with commit ID - 31e0f29a6140a19eda8de615e615208f51f2cf96 > Allocate memory for value vectors upfront in flatten operator > - > > Key: DRILL-6126 > URL: https://issues.apache.org/jira/browse/DRILL-6126 > Project: Apache Drill > Issue Type: Improvement >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Critical > Labels: ready-to-commit > Fix For: 1.13.0 > > > With recent changes to control batch size for flatten operator, we figure out > row count in the output batch based on memory. Since we know how many rows we > are going to include in the batch, we can also allocate the memory needed > upfront instead of starting with initial value (4096) and doubling, copying > every time we need more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6216) Metadata mismatch when connecting to a Drill 1.12.0 with a Drill-1.13.0-SNAPSHOT driver
[
https://issues.apache.org/jira/browse/DRILL-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395696#comment-16395696
]
Boaz Ben-Zvi commented on DRILL-6216:
-
Merged with commit ID - b4a783a0b0fb334134fa00411cb4b910c05baa05
> Metadata mismatch when connecting to a Drill 1.12.0 with a
> Drill-1.13.0-SNAPSHOT driver
> ---
>
> Key: DRILL-6216
> URL: https://issues.apache.org/jira/browse/DRILL-6216
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.13.0
>Reporter: Kunal Khatua
>Assignee: Sorabh Hamirwasia
>Priority: Blocker
> Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> When running against a 1.12.0 Drill server
> {code:sql}
> select * from sys.`options`;
> {code}
> I hit the following issue in Squirrel-SQL client with the 1.13.0-SNAPSHOT
> JDBC driver.
> {code:java}
> 2018-02-20 11:41:16,557 [Thread-2] DEBUG
> net.sourceforge.squirrel_sql.fw.util.DefaultExceptionFormatter - Error
> java.sql.SQLException: Unexpected RuntimeException:
> java.lang.IllegalArgumentException: The field [`$values$` (BIGINT:OPTIONAL)]
> doesn't match the provided metadata major_type {
> minor_type: BIGINT
> mode: OPTIONAL
> }
> name_part {
> name: "num_val"
> }
> value_count: 153
> buffer_length: 1224
> .
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:544)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:602)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1918)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:66)
> at
> oadd.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:630)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1106)
> at
> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1117)
> at
> oadd.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:638)
> at
> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:193)
> at
> oadd.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:149)
> at
> oadd.org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:209)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:101)
> at
> net.sourceforge.squirrel_sql.client.session.StatementWrapper.execute(StatementWrapper.java:165)
> at
> net.sourceforge.squirrel_sql.client.session.SQLExecuterTask.processQuery(SQLExecuterTask.java:378)
> at
> net.sourceforge.squirrel_sql.client.session.SQLExecuterTask.run(SQLExecuterTask.java:221)
> at
> net.sourceforge.squirrel_sql.fw.util.TaskExecuter.run(TaskExecuter.java:82)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.IllegalArgumentException: The field [`$values$`
> (BIGINT:OPTIONAL)] doesn't match the provided metadata major_type {
> minor_type: BIGINT
> mode: OPTIONAL
> }
> name_part {
> name: "num_val"
> }
> value_count: 153
> buffer_length: 1224
> .
> at
> oadd.com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
> at
> oadd.org.apache.drill.exec.vector.BigIntVector.load(BigIntVector.java:299)
> at
> oadd.org.apache.drill.exec.vector.NullableBigIntVector.load(NullableBigIntVector.java:295)
> at
> oadd.org.apache.drill.exec.record.RecordBatchLoader.load(RecordBatchLoader.java:135)
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:506)
> ... 16 more
> {code}
> This issue however, doesn't exist when working with Apache Master
> 1.13.0-SNAPSHOT
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6210) Enhance the test schema builder for remaining types
[
https://issues.apache.org/jira/browse/DRILL-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395692#comment-16395692
]
Boaz Ben-Zvi commented on DRILL-6210:
-
Merged with commit ID - f653359c7a5468ef94815e5d845980d254cda7b4
> Enhance the test schema builder for remaining types
> ---
>
> Key: DRILL-6210
> URL: https://issues.apache.org/jira/browse/DRILL-6210
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> The result set loader project enhanced the schema builder used in tests to
> handle Drill's complex types: maps, unions, lists and repeated lists. The
> schema builder previously only handled maps.
> This ticket describes the addition of just this one part of the result set
> loader.
> Also adds a run-time schema improvement: functions to create arrays when
> testing rather than writing out the {{new String[] \{"foo", "bar"\}}} syntax.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6145) Implement Hive MapR-DB JSON handler.
[
https://issues.apache.org/jira/browse/DRILL-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395668#comment-16395668
]
ASF GitHub Bot commented on DRILL-6145:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1158#discussion_r173901690
--- Diff: distribution/pom.xml ---
@@ -324,6 +324,14 @@
org.apache.hbase
hbase-protocol
+
--- End diff --
Should the dependency be part of hive module (even if it is not part of
compile and is a run-time dependency)?
> Implement Hive MapR-DB JSON handler.
> -
>
> Key: DRILL-6145
> URL: https://issues.apache.org/jira/browse/DRILL-6145
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive, Storage - MapRDB
>Affects Versions: 1.12.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> Similar to "hive-hbase-storage-handler" to support querying MapR-DB Hive's
> external tables it is necessary to add "hive-maprdb-json-handler".
> Use case:
> # Create a table MapR-DB JSON table:
> {code}
> _> mapr dbshell_
> _maprdb root:> create /tmp/table/json_ (make sure /tmp/table exists)
> {code}
> -- insert data
> {code}
> insert /tmp/table/json --value '\{"_id":"movie002" , "title":"Developers
> on the Edge", "studio":"Command Line Studios"}'
> insert /tmp/table/json --id movie003 --value '\{"title":"The Golden
> Master", "studio":"All-Nighter"}'
> {code}
> # Create a Hive external table:
> {code}
> hive> CREATE EXTERNAL TABLE mapr_db_json_hive_tbl (
> > movie_id string, title string, studio string)
> > STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
> > TBLPROPERTIES("maprdb.table.name" =
> "/tmp/table/json","maprdb.column.id" = "movie_id");
> {code}
>
> # Use hive schema to query this table via Drill:
> {code}
> 0: jdbc:drill:> select * from hive.mapr_db_json_hive_tbl;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6234) VarCharVector setValueCount can throw out of memory exception
Timothy Farkas created DRILL-6234:
-
Summary: VarCharVector setValueCount can throw out of memory
exception
Key: DRILL-6234
URL: https://issues.apache.org/jira/browse/DRILL-6234
Project: Apache Drill
Issue Type: Improvement
Reporter: Timothy Farkas
Assignee: Timothy Farkas
Doing the following will throw an Index out of bounds exception.
{code}
final VarCharVector vector = new VarCharVector(field, allocator);
vector.allocateNew();
vector.getMutator().setValueCount(100);
{code}
The expected behavior is to resize the array appropriately. If an index is
uninitialized it can return an empty byte array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-4847) Window function query results in OOM Exception.
[
https://issues.apache.org/jira/browse/DRILL-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker reassigned DRILL-4847:
Assignee: Pritesh Maker (was: Paul Rogers)
> Window function query results in OOM Exception.
> ---
>
> Key: DRILL-4847
> URL: https://issues.apache.org/jira/browse/DRILL-4847
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.8.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Pritesh Maker
>Priority: Critical
> Labels: window_function
> Attachments: drillbit.log
>
>
> Window function query results in OOM Exception.
> Drill version 1.8.0-SNAPSHOT git commit ID: 38ce31ca
> MapRBuildVersion 5.1.0.37549.GA
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> SELECT clientname, audiencekey, spendprofileid,
> postalcd, provincecd, provincename, postalcode_json, country_json,
> province_json, town_json, dma_json, msa_json, ROW_NUMBER() OVER (PARTITION BY
> spendprofileid ORDER BY (CASE WHEN postalcd IS NULL THEN 9 ELSE 0 END) ASC,
> provincecd ASC) as rn FROM `MD593.parquet` limit 3;
> Error: RESOURCE ERROR: One or more nodes ran out of memory while executing
> the query.
> Failure while allocating buffer.
> Fragment 0:0
> [Error Id: 2287fe71-f0cb-469a-a563-11580fceb1c5 on centos-01.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2016-08-16 07:25:44,590 [284d4006-9f9d-b893-9352-4f54f9b1d52a:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 284d4006-9f9d-b893-9352-4f54f9b1d52a: SELECT clientname, audiencekey,
> spendprofileid, postalcd, provincecd, provincename, postalcode_json,
> country_json, province_json, town_json, dma_json, msa_json, ROW_NUMBER() OVER
> (PARTITION BY spendprofileid ORDER BY (CASE WHEN postalcd IS NULL THEN 9
> ELSE 0 END) ASC, provincecd ASC) as rn FROM `MD593.parquet` limit 3
> ...
> 2016-08-16 07:25:46,273 [284d4006-9f9d-b893-9352-4f54f9b1d52a:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/284d4006-9f9d-b893-9352-4f54f9b1d52a_majorfragment0_minorfragment0_operator8/2
> 2016-08-16 07:25:46,283 [284d4006-9f9d-b893-9352-4f54f9b1d52a:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
> nodes ran out of memory while executing the query.
> Failure while allocating buffer.
> [Error Id: 2287fe71-f0cb-469a-a563-11580fceb1c5 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
> ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:242)
> [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101]
> Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure
> while allocating buffer.
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:187)
> ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapTransferPair.(RepeatedMapVector.java:331)
> ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapTransferPair.(RepeatedMapVector.java:307)
> ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.RepeatedMapVector.getTransferPair(RepeatedMapVector.java:161)
> ~[vector-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.SimpleVectorWrapper.cloneAndTransfer(SimpleVectorWrapper.java:66)
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.VectorContainer.cloneAndTransfer(VectorContainer.java:204)
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.VectorContainer.getTransferClone(VectorContainer.java:157)
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:569)
> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort
[jira] [Commented] (DRILL-6125) PartitionSenderRootExec can leak memory because close method is not synchronized
[ https://issues.apache.org/jira/browse/DRILL-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395574#comment-16395574 ] ASF GitHub Bot commented on DRILL-6125: --- Github user ilooner commented on the issue: https://github.com/apache/drill/pull/1105 Squashed commits. @arina-ielchiieva please let me know if you have any comments. > PartitionSenderRootExec can leak memory because close method is not > synchronized > > > Key: DRILL-6125 > URL: https://issues.apache.org/jira/browse/DRILL-6125 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.13.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Minor > Fix For: 1.14.0 > > > PartitionSenderRootExec creates a PartitionerDecorator and saves it in the > *partitioner* field. The creation of the partitioner happens in the > createPartitioner method. This method get's called by the main fragment > thread. The partitioner field is accessed by the fragment thread during > normal execution but it can also be accessed by the receivingFragmentFinished > method which is a callback executed by the event processor thread. Because > multiple threads can access the partitioner field synchronization is done on > creation and on when receivingFragmentFinished. However, the close method can > also be called by the event processor thread, and the close method does not > synchronize before accessing the partitioner field. Since synchronization is > not done the event processor thread may have an old reference to the > partitioner when a query cancellation is done. Since it has an old reference > the current partitioner can may not be cleared and a memory leak may occur. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395567#comment-16395567
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/1160
I think I need to put up some tooltips.
We're getting the metrics from the `drill.allocator.root.[peak|used]`
values. I haven't seen a metric that shows the actual Direct allocation from
the system. What Drill seems to report is the current **actively** used Direct
memory. So, when running queries on the cluster, these numbers move up. The
percentage for Direct is the current usage as a percent of the peak memory that
was reported by the root allocator.
Check out any active Drillbit's `#metrics` page. The `drill.allocator.*`
entries are what I was referring to. JVM itself doesn't appear to expose the
Direct memory information readily.
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on
[
https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6187:
-
Labels: ready-to-commit (was: )
> Exception in RPC communication between DataClient/ControlClient and
> respective servers when bit-to-bit security is on
> -
>
> Key: DRILL-6187
> URL: https://issues.apache.org/jira/browse/DRILL-6187
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC, Security
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.13.0
>
>
>
> {color:#00}Below is the summary of issue: {color}
>
> {color:#00}*Scenario:*{color}
> {color:#00}It seems like first sendRecordBatch was sent to Foreman which
> initiated the Authentication handshake. But before initiating handshake for
> auth we establish a connection and store that in a registry. Now if in
> parallel there is another recordBatch (by a different minor fragment running
> on same Drillbit) to be sent then that will see the connection available in
> registry and will initiate the send. Before the authentication is completed
> this second request reached foreman and it throws below exception saying RPC
> type 3 message is not allowed and closes the connection. This also fails the
> authentication handshake which was in progress.{color}{color:#00} Here
> the logs with details:{color}
> {color:#00} {color}
> {color:#00}*Forman received the SASL_START message from another
> node:*{color}
> {color:#00}*_2018-02-21 18:43:30,759
> [_*{color}{color:#00}_BitServer-4] TRACE
> o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START
> from /10.10.100.161:35482_{color}
> {color:#00} {color}
> {color:#00}*Then around same time it received another message from client
> of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not
> completed yet.*{color}
> {color:#00} {color}
> {color:#00}*_2018-02-21 18:43:30,762_*{color}{color:#00}
> _[BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC
> communication. Connection: /10.10.100.162:31012 <--> /__10.10.100.161:35482_
> _(data server). Closing connection._{color}
> {color:#00}_io.netty.handler.codec.DecoderException:
> org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed
> without authentication. Client on /__10.10.100.161:35482_ _must authenticate
> before making requests. Connection dropped. [Details: Encryption: enabled ,
> MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#00} {color}
> {color:#00}*Then client receives an channel closed exception:*{color}
> {color:#00} {color}
> {color:#00}*2018-02-21 18:43:30,764 [*{color}{color:#00}BitClient-4]
> WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed
> channel. Connection: /_10.10.100.161:35482_ <--> _10.10.100.162:31012_ (data
> client){color}
> {color:#00} {color}
> {color:#00}*and due to this it's initial command for authentication also
> fails. Since there is channel closed exception above I will think that
> triggered the failure of authentication request as well.*{color}
> {color:#00} {color}
> {color:#00}_Caused by: org.apache.drill.exec.rpc.RpcException: Command
> failed while establishing connection. Failure type AUTHENTICATION._{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_
[jira] [Created] (DRILL-6233) Flattening an Array column (JSON file) return no rows
Volodymyr Tkach created DRILL-6233:
--
Summary: Flattening an Array column (JSON file) return no rows
Key: DRILL-6233
URL: https://issues.apache.org/jira/browse/DRILL-6233
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Volodymyr Tkach
Assignee: Volodymyr Tkach
Flatten return no rows if array is empty.
{noformat}
flat.json
{
"x" : 5,
"y" : "a string",
"z" : []
}
{noformat}
folowing query returns no results:
{noformat}
0: jdbc:drill:zk=local> select flatten(z) from dfs.tmp.`flat.json`;
+-+
| EXPR$0 |
+-+
+-+
{noformat}
but has to return one row with null for flatten function
{noformat}
++---+-+
| x | y | z | EXPR$1 |
++---+-+
| 5 | a string | [] | null |
++---+-+
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-1491) Support for JDK 8
[ https://issues.apache.org/jira/browse/DRILL-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395122#comment-16395122 ] ASF GitHub Bot commented on DRILL-1491: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1143 > Support for JDK 8 > - > > Key: DRILL-1491 > URL: https://issues.apache.org/jira/browse/DRILL-1491 > Project: Apache Drill > Issue Type: Task > Components: Tools, Build & Test >Reporter: Aditya Kishore >Assignee: Volodymyr Tkach >Priority: Blocker > Fix For: 1.13.0 > > Attachments: DRILL-1491.1.patch.txt > > > This will be the umbrella JIRA used to track and fix issues with JDK 8 > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on
[
https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395123#comment-16395123
]
ASF GitHub Bot commented on DRILL-6187:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1145
> Exception in RPC communication between DataClient/ControlClient and
> respective servers when bit-to-bit security is on
> -
>
> Key: DRILL-6187
> URL: https://issues.apache.org/jira/browse/DRILL-6187
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC, Security
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.13.0
>
>
>
> {color:#00}Below is the summary of issue: {color}
>
> {color:#00}*Scenario:*{color}
> {color:#00}It seems like first sendRecordBatch was sent to Foreman which
> initiated the Authentication handshake. But before initiating handshake for
> auth we establish a connection and store that in a registry. Now if in
> parallel there is another recordBatch (by a different minor fragment running
> on same Drillbit) to be sent then that will see the connection available in
> registry and will initiate the send. Before the authentication is completed
> this second request reached foreman and it throws below exception saying RPC
> type 3 message is not allowed and closes the connection. This also fails the
> authentication handshake which was in progress.{color}{color:#00} Here
> the logs with details:{color}
> {color:#00} {color}
> {color:#00}*Forman received the SASL_START message from another
> node:*{color}
> {color:#00}*_2018-02-21 18:43:30,759
> [_*{color}{color:#00}_BitServer-4] TRACE
> o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START
> from /10.10.100.161:35482_{color}
> {color:#00} {color}
> {color:#00}*Then around same time it received another message from client
> of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not
> completed yet.*{color}
> {color:#00} {color}
> {color:#00}*_2018-02-21 18:43:30,762_*{color}{color:#00}
> _[BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC
> communication. Connection: /10.10.100.162:31012 <--> /__10.10.100.161:35482_
> _(data server). Closing connection._{color}
> {color:#00}_io.netty.handler.codec.DecoderException:
> org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed
> without authentication. Client on /__10.10.100.161:35482_ _must authenticate
> before making requests. Connection dropped. [Details: Encryption: enabled ,
> MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#00} {color}
> {color:#00}*Then client receives an channel closed exception:*{color}
> {color:#00} {color}
> {color:#00}*2018-02-21 18:43:30,764 [*{color}{color:#00}BitClient-4]
> WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed
> channel. Connection: /_10.10.100.161:35482_ <--> _10.10.100.162:31012_ (data
> client){color}
> {color:#00} {color}
> {color:#00}*and due to this it's initial command for authentication also
> fails. Since there is channel closed exception above I will think that
> triggered the failure of authentication request as well.*{color}
> {color:#00} {color}
> {color:#00}_Caused by: org.apache.drill.exec.rpc.RpcException: Command
> failed while establishing connection. Failure type AUTHENTICATION._{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(Recon
[jira] [Commented] (DRILL-6044) Shutdown button does not work from WebUI
[ https://issues.apache.org/jira/browse/DRILL-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395124#comment-16395124 ] ASF GitHub Bot commented on DRILL-6044: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/1153 > Shutdown button does not work from WebUI > > > Key: DRILL-6044 > URL: https://issues.apache.org/jira/browse/DRILL-6044 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 1.13.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Critical > Labels: ready-to-commit > Fix For: 1.13.0 > > Attachments: Screen Shot 2017-12-19 at 10.51.16 AM.png > > > git.commit.id.abbrev=eb0c403 > Nothing happens when click on the SHUTDOWN button from the WebUI. The > browser's debugger showed that the request failed due to access control > checks (see attached screen shot). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4547) Javadoc fails with Java8
[
https://issues.apache.org/jira/browse/DRILL-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395116#comment-16395116
]
Parth Chandra commented on DRILL-4547:
--
Disabling doclint for release 1.13, which will allow the build to go thru. [ in
commit 035010c].
> Javadoc fails with Java8
>
>
> Key: DRILL-4547
> URL: https://issues.apache.org/jira/browse/DRILL-4547
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Affects Versions: 1.6.0
>Reporter: Laurent Goujon
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
>
> Javadoc cannot be generated when using Java8 (likely because the parser is
> now more strict).
> Here's an example of issues when trying to generate javadocs in module
> {{drill-fmpp-maven-plugin}}
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:2.9.1:jar (attach-javadocs) on
> project drill-fmpp-maven-plugin: MavenReportException: Error while creating
> archive:
> [ERROR] Exit code: 1 -
> /Users/laurent/devel/drill/tools/fmpp/src/main/java/org/apache/drill/fmpp/mojo/FMPPMojo.java:44:
> error: unknown tag: goal
> [ERROR] * @goal generate
> [ERROR] ^
> [ERROR]
> /Users/laurent/devel/drill/tools/fmpp/src/main/java/org/apache/drill/fmpp/mojo/FMPPMojo.java:45:
> error: unknown tag: phase
> [ERROR] * @phase generate-sources
> [ERROR] ^
> [ERROR]
> /Users/laurent/devel/drill/tools/fmpp/target/generated-sources/plugin/org/apache/drill/fmpp/mojo/HelpMojo.java:25:
> error: unknown tag: goal
> [ERROR] * @goal help
> [ERROR] ^
> [ERROR]
> /Users/laurent/devel/drill/tools/fmpp/target/generated-sources/plugin/org/apache/drill/fmpp/mojo/HelpMojo.java:26:
> error: unknown tag: requiresProject
> [ERROR] * @requiresProject false
> [ERROR] ^
> [ERROR]
> /Users/laurent/devel/drill/tools/fmpp/target/generated-sources/plugin/org/apache/drill/fmpp/mojo/HelpMojo.java:27:
> error: unknown tag: threadSafe
> [ERROR] * @threadSafe
> [ERROR] ^
> [ERROR]
> [ERROR] Command line was:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_72.jdk/Contents/Home/bin/javadoc
> @options @packages
> [ERROR]
> [ERROR] Refer to the generated Javadoc files in
> '/Users/laurent/devel/drill/tools/fmpp/target/apidocs' dir.
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :drill-fmpp-maven-plugin
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-1491) Support for JDK 8
[ https://issues.apache.org/jira/browse/DRILL-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395117#comment-16395117 ] ASF GitHub Bot commented on DRILL-1491: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/1143 +1 > Support for JDK 8 > - > > Key: DRILL-1491 > URL: https://issues.apache.org/jira/browse/DRILL-1491 > Project: Apache Drill > Issue Type: Task > Components: Tools, Build & Test >Reporter: Aditya Kishore >Assignee: Volodymyr Tkach >Priority: Blocker > Fix For: 1.13.0 > > Attachments: DRILL-1491.1.patch.txt > > > This will be the umbrella JIRA used to track and fix issues with JDK 8 > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6044) Shutdown button does not work from WebUI
[ https://issues.apache.org/jira/browse/DRILL-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395109#comment-16395109 ] ASF GitHub Bot commented on DRILL-6044: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/1153 +1 > Shutdown button does not work from WebUI > > > Key: DRILL-6044 > URL: https://issues.apache.org/jira/browse/DRILL-6044 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 1.13.0 >Reporter: Krystal >Assignee: Venkata Jyothsna Donapati >Priority: Critical > Labels: ready-to-commit > Fix For: 1.13.0 > > Attachments: Screen Shot 2017-12-19 at 10.51.16 AM.png > > > git.commit.id.abbrev=eb0c403 > Nothing happens when click on the SHUTDOWN button from the WebUI. The > browser's debugger showed that the request failed due to access control > checks (see attached screen shot). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on
[
https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395110#comment-16395110
]
ASF GitHub Bot commented on DRILL-6187:
---
Github user parthchandra commented on the issue:
https://github.com/apache/drill/pull/1145
+1
> Exception in RPC communication between DataClient/ControlClient and
> respective servers when bit-to-bit security is on
> -
>
> Key: DRILL-6187
> URL: https://issues.apache.org/jira/browse/DRILL-6187
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC, Security
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.13.0
>
>
>
> {color:#00}Below is the summary of issue: {color}
>
> {color:#00}*Scenario:*{color}
> {color:#00}It seems like first sendRecordBatch was sent to Foreman which
> initiated the Authentication handshake. But before initiating handshake for
> auth we establish a connection and store that in a registry. Now if in
> parallel there is another recordBatch (by a different minor fragment running
> on same Drillbit) to be sent then that will see the connection available in
> registry and will initiate the send. Before the authentication is completed
> this second request reached foreman and it throws below exception saying RPC
> type 3 message is not allowed and closes the connection. This also fails the
> authentication handshake which was in progress.{color}{color:#00} Here
> the logs with details:{color}
> {color:#00} {color}
> {color:#00}*Forman received the SASL_START message from another
> node:*{color}
> {color:#00}*_2018-02-21 18:43:30,759
> [_*{color}{color:#00}_BitServer-4] TRACE
> o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START
> from /10.10.100.161:35482_{color}
> {color:#00} {color}
> {color:#00}*Then around same time it received another message from client
> of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not
> completed yet.*{color}
> {color:#00} {color}
> {color:#00}*_2018-02-21 18:43:30,762_*{color}{color:#00}
> _[BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC
> communication. Connection: /10.10.100.162:31012 <--> /__10.10.100.161:35482_
> _(data server). Closing connection._{color}
> {color:#00}_io.netty.handler.codec.DecoderException:
> org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed
> without authentication. Client on /__10.10.100.161:35482_ _must authenticate
> before making requests. Connection dropped. [Details: Encryption: enabled ,
> MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#00} {color}
> {color:#00}*Then client receives an channel closed exception:*{color}
> {color:#00} {color}
> {color:#00}*2018-02-21 18:43:30,764 [*{color}{color:#00}BitClient-4]
> WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed
> channel. Connection: /_10.10.100.161:35482_ <--> _10.10.100.162:31012_ (data
> client){color}
> {color:#00} {color}
> {color:#00}*and due to this it's initial command for authentication also
> fails. Since there is channel closed exception above I will think that
> triggered the failure of authentication request as well.*{color}
> {color:#00} {color}
> {color:#00}_Caused by: org.apache.drill.exec.rpc.RpcException: Command
> failed while establishing connection. Failure type AUTHENTICATION._{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
> ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00} _at
> org.apache.drill.exec.rpc.ReconnectingConnection.runC
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395061#comment-16395061
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user parthchandra commented on the issue:
https://github.com/apache/drill/pull/1160
Looks wrong dunnit? Direct memory is (still?) showing as zero?
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6217) NaN/Inf: NestedLoopJoin processes NaN values incorrectly
[
https://issues.apache.org/jira/browse/DRILL-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394990#comment-16394990
]
Arina Ielchiieva commented on DRILL-6217:
-
Merged with commit id d09efb931f183d989516d1dd098ade546bbe3f16.
> NaN/Inf: NestedLoopJoin processes NaN values incorrectly
>
>
> Key: DRILL-6217
> URL: https://issues.apache.org/jira/browse/DRILL-6217
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.13.0
>
> Attachments: ObjsX.json
>
>
> *AFFECTED_FUNCTIONALITY:* INNER JOIN (nestedloopjoin)
> *ISSUE_DESCRIPTION:* according to *nestedloopjoin* query result NaN != NaN,
> however hashjoin / mergejoin behaves another way - NaN = NaN. As far as I
> understand, nestedloopjoin should behave like hashjoin / mergejoin. *STEPS*:
> - Upload the attached file to Hadoop fs (ObjsX.json);
> - Setup the following system settings:
> *set planner.enable_nljoin_for_scalar_only = false*
> *set planner.enable_hashjoin = false*
> *set planner.enable_mergejoin = false*
> *set planner.enable_nestedloopjoin = true*
> - Run the following sql query {code} select distinct t.name from
> dfs.tmp.`ObjsX.json` t inner join dfs.tmp.`ObjsX.json` tt on t.attr4 =
> tt.attr4 {code}
> *EXPECTED_RESULT:* It was expected to get the following result:
> {code}
>object1
>object2
>object3
>object4
> {code}
>
> *ACTUAL_RESULT:* The actual result is:
> {code}
>object2
>object3
>object4
> {code}
> Please investigate and fix
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6213) During restart drillbit should be killed forcefully if exceeds allowed timeout
[ https://issues.apache.org/jira/browse/DRILL-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394989#comment-16394989 ] Arina Ielchiieva commented on DRILL-6213: - Merged with commit id f47af650bf4f2cc1c61aff26bac92dec0674d0d4. > During restart drillbit should be killed forcefully if exceeds allowed timeout > -- > > Key: DRILL-6213 > URL: https://issues.apache.org/jira/browse/DRILL-6213 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.12.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > > drillbit.sh can start / stop drillbit. Stop command will stop drillbit > softly. If drillbit is not stopped during pre-defined time, it it will be > killed forcefully (-9). > There is also restart command which basically behaves like stop + start. But > in this case stop command is called without kill forcefully flag. This leads > to the cases when executing restart it hangs since driilbit is not be able to > be killed softly and retries to do it forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6192) Drill is vulnerable to CVE-2017-12197
[ https://issues.apache.org/jira/browse/DRILL-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394988#comment-16394988 ] Arina Ielchiieva commented on DRILL-6192: - Merged with commit id b75298d178bf54e8758070de87e456e620ba6811. > Drill is vulnerable to CVE-2017-12197 > - > > Key: DRILL-6192 > URL: https://issues.apache.org/jira/browse/DRILL-6192 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Volodymyr Tkach >Assignee: Volodymyr Tkach >Priority: Major > Labels: ready-to-commit > Fix For: 1.13.0 > > > The current version of libpam4j bundled with MCS does not perform any > authorization check. Any user with valid password could access the cluster > even if the user account is disabled/password expired/'not allowed to access > the service(pam_access ..)' etc.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6225) Add support for boost 1.63+ with openSSL 1.1.0 support
[ https://issues.apache.org/jira/browse/DRILL-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394918#comment-16394918 ] Rob Wu commented on DRILL-6225: --- Thanks [~parthc], you are right and also 1.0.2 EOL seems to be Eo2019 so I think we can treat this ticket as a reminder item for now. Thanks! > Add support for boost 1.63+ with openSSL 1.1.0 support > -- > > Key: DRILL-6225 > URL: https://issues.apache.org/jira/browse/DRILL-6225 > Project: Apache Drill > Issue Type: Task > Components: Client - C++ >Reporter: Rob Wu >Priority: Minor > > Boost 1.57 is not able to compile with openSSL 1.1 > ([https://svn.boost.org/trac10/ticket/12238)] and adding > add_definitions(-DOPENSSL_API_COMPAT=0x1000L) does not work. > > In order to add support for openSSL 1.1, we would need to upgrade boost to > 1.62+. However, it looks like boost 1.62 bcp will segfault on asio component > when you attempt to shade the boost libraries > ([https://svn.boost.org/trac10/ticket/12357)]. So in that case, we should > upgrade to 1.63+. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
