[jira] [Commented] (DRILL-6361) Provide sqlTypeOf() and modeOf() functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459458#comment-16459458
 ] 

ASF GitHub Bot commented on DRILL-6361:
---

Github user kfaraaz commented on the issue:

https://github.com/apache/drill/pull/1242
  
Can you please add a unit test that covers these types, BINARY, BOOLEAN, 
DECIMAL, DATE, TIME and TIMESTAMP ?


> Provide sqlTypeOf() and modeOf() functions
> --
>
> Key: DRILL-6361
> URL: https://issues.apache.org/jira/browse/DRILL-6361
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>  Labels: doc-impacting
>
> Drill provides a {{typeof()}} function to return the type of a column. The 
> returned string, however, has only the base data type. A Drill data type (a 
> "major type") also includes a cardinality (a "mode"). For example, {{Optional 
> Int}} or {{Required VarChar}}.
> This type information is useful for handling data conversions. For example, 
> if I could tell that a column value was a {{Nullable Int}}, I could guess 
> that it is one Drill invented, and I could merge it, by hand, with the type 
> from another file that had actual values.
> The two options are equivalent. Either provide a {{modeOf()}} to just return 
> cardinality, or a {{dataTypeOf()}} that returns both. (Maybe the {{modeOf()}} 
> might be more useful.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (DRILL-5904) TestCTAS.testPartitionByForAllType fails sporadically on some build machines

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas closed DRILL-5904.
-
Resolution: Cannot Reproduce

> TestCTAS.testPartitionByForAllType fails sporadically on some build machines
> 
>
> Key: DRILL-5904
> URL: https://issues.apache.org/jira/browse/DRILL-5904
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> Vlad found that the TestCTAS.testPartitionByForAllType test sporadically 
> fails with this stack trace sometimes.
> testPartitionByForAllTypes(org.apache.drill.exec.sql.TestCTAS)
> java.lang.Exception: test timed out after 10 milliseconds
>   at java.io.UnixFileSystem.canonicalize0(Native Method) ~[na:1.7.0_131]
>   at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172) 
> ~[na:1.7.0_131]
>   at java.io.File.getCanonicalPath(File.java:618) ~[na:1.7.0_131]
>   at java.io.File.getCanonicalFile(File.java:643) ~[na:1.7.0_131]
>   at org.apache.commons.io.FileUtils.isSymlink(FileUtils.java:2935) 
> ~[commons-io-2.4.jar:2.4]
>   at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1534) 
> ~[commons-io-2.4.jar:2.4]
>   at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270) 
> ~[commons-io-2.4.jar:2.4]
>   at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653) 
> ~[commons-io-2.4.jar:2.4]
>   at org.apache.commons.io.FileUtils.deleteQuietly(FileUtils.java:1566) 
> ~[commons-io-2.4.jar:2.4]
>   at 
> org.apache.drill.exec.sql.TestCTAS.testPartitionByForAllTypes(TestCTAS.java:292)
>  ~[test-classes/:na]
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
> ~[na:1.7.0_131]
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
> ~[na:1.7.0_131]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.7.0_131]
>   at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>  ~[junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  ~[junit-4.11.jar:na]
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>  ~[junit-4.11.jar:na]
>   at 
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
>  ~[jmockit-1.3.jar:na]
>   at 
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
>  ~[jmockit-1.3.jar:na]
>   at 
> mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
>  ~[jmockit-1.3.jar:na]
>   at sun.reflect.GeneratedMethodAccessor97.invoke(Unknown Source) ~[na:na]
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  ~[na:1.7.0_131]
>   at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_131]
>   at 
> mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
>  ~[jmockit-1.3.jar:na]
>   at 
> mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
>  ~[jmockit-1.3.jar:na]
>   at 
> mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41) 
> ~[jmockit-1.3.jar:na]
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
>  ~[junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  ~[junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
> ~[junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 
> ~[junit-4.11.jar:na]
>   at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
>  ~[junit-4.11.jar:na]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-6352) Investigate IndexOutOfBoundsException in TestBsonRecordReader

[Timothy Farkas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timothy Farkas resolved DRILL-6352.
---
Resolution: Fixed

> Investigate IndexOutOfBoundsException in TestBsonRecordReader
> -
>
> Key: DRILL-6352
> URL: https://issues.apache.org/jira/browse/DRILL-6352
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
>
> TestBsonRecordReader requires 400kb on the allocator in order to run all 
> tests successfully. Reducing the memory below that to 300kb will cause an IOB
> {code}
> objc[92518]: Class JavaLaunchHelper is implemented in both 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/bin/java 
> (0x10e8fe4c0) and 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/libinstrument.dylib
>  (0x10e9824e0). One of the two will be used. Which one is undefined.
> java.lang.IndexOutOfBoundsException: DrillBuf[7], udle: [1 0..0], index: 0, 
> length: 4 (expected: range(0, 0))
> DrillBuf[7], udle: [1 0..0]
>   at 
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
>   at 
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
>   at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114)
>   at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484)
>   at 
> org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:696)
>   at 
> org.apache.drill.exec.vector.NullableVarCharVector$Mutator.setSafe(NullableVarCharVector.java:609)
>   at 
> org.apache.drill.exec.vector.complex.impl.NullableVarCharWriterImpl.write(NullableVarCharWriterImpl.java:110)
>   at 
> org.apache.drill.exec.store.bson.BsonRecordReader.writeString(BsonRecordReader.java:276)
>   at 
> org.apache.drill.exec.store.bson.BsonRecordReader.writeToListOrMap(BsonRecordReader.java:167)
>   at 
> org.apache.drill.exec.store.bson.BsonRecordReader.writeToListOrMap(BsonRecordReader.java:139)
>   at 
> org.apache.drill.exec.store.bson.BsonRecordReader.write(BsonRecordReader.java:75)
>   at 
> org.apache.drill.exec.store.bson.TestBsonRecordReader.testRecursiveDocuments(TestBsonRecordReader.java:193)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>   at 
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459411#comment-16459411
 ] 

ASF GitHub Bot commented on DRILL-6321:
---

Github user chunhui-shi commented on a diff in the pull request:

https://github.com/apache/drill/pull/1224#discussion_r185159367
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConformance.java
 ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql;
+
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlDelegatingConformance;
+
+/**
+ * Drill's SQL conformance is SqlConformanceEnum.DEFAULT except for method 
isApplyAllowed().
+ * Since Drill is going to allow OUTER APPLY and CROSS APPLY to allow each 
row from left child of Join
+ * to join with output of right side (sub-query or table function that 
will be invoked for each row).
+ * Refer to DRILL-5999 for more information.
+ */
+public class DrillConformance extends SqlDelegatingConformance {
--- End diff --

I think DrillConformance should be an independent class since we rely on 
this class to define the conformance used in Drill and we might add something 
more in the future to enable some other syntax. Let me know if this answer your 
suggestion.


> Lateral Join: Planning changes - enable submitting physical plan
> 
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
>
> Implement changes to enable submitting a physical plan containing lateral and 
> unnest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459404#comment-16459404
 ] 

ASF GitHub Bot commented on DRILL-6373:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/1244
  
@ppadma, here is the next step in our long saga to check in the batch 
sizing work. Please give it a review at your convenience. Thanks! 


> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the 
> {{ResultSetLoader}} and related classes to prepare for Union and List 
> support. This fix follows the refactoring of the column accessors for the 
> same purpose. Actual Union and List support is to follow in a separate PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459402#comment-16459402
 ] 

ASF GitHub Bot commented on DRILL-6373:
---

GitHub user paul-rogers opened a pull request:

https://github.com/apache/drill/pull/1244

DRILL-6373: Refactor Result Set Loader for Union, List support

This PR builds on the previous refactoring of the column accessors to 
prepare for Union, (non-repeated) List and Repeated List support. The PR 
includes four closely related changes divided across four commits:

### Correct the Type of the Data Vector in a Nullable Vector

The nullable vectors contain a "bits" vector and a "data" vector. The data 
vector has historically been created using the same `MaterializedField` as the 
nullable vector, meaning that the data vector is labeled as "nullable" even 
though it has no bits vector.

This PR creates a clone `MaterializedField` with the same name as the outer 
nullable vector, but with a Required type.

This change ensures that the overflow logic works correctly as it uses the 
vector metadata (in the `MaterializedField`) to know what kind of vector to 
create for the "lookahead" vector.

### Result Set Loader Refactor

The second commit pretty much just rearranges the deck chairs in a way that 
we an slot in the new types in the next PR. The need for the changes can be 
seen in the full code set (the union and list support was pulled out for this 
PR.)

A union is a container, like a map, so the tuple state was refactored to 
create a common parent container state.

List and unions are very complex to build, so the code to build the 
internal workings of each vector was pulled out into a separate builder class.

### Projection Handling and the Vector Cache

Previous versions of the result set loader handled projection and a cache 
for vectors reused across readers in the same Scan operator. Once we introduce 
nested maps, projection within maps, unions and lists, projection gets much 
more complex, as does vector caching.

This PR adds logic to support projection and vector caching to any 
arbitrary level of maps. It turns out that handling projection of an entire 
map, and projection of fields within maps, is far more complex than you'd 
think, requiring quite a bit of internal state to keep everything straight. The 
result is that we can now handle a map `m` with three fields `{a, b, c}` and 
project just one of them, `m.a`, say.

Further, Drill allows projection of non-existent columns. So, we might ask 
for field `m.d` which does not exist in the above map. The projection mechanism 
handles this case as well, creating the right kind of null column.

### Unit Tests

New tests are added to exercise the projection and cache mechanisms. 
Existing tests were updated for the changes made in the refactoring.

### Reference Design

All of this work is done in support of the overall "batch sizing" project 
explained 
[here](https://github.com/paul-rogers/drill/wiki/Batch-Handling-Upgrades).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/paul-rogers/drill DRILL-6373

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1244.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1244


commit 7df4280f3011862b84b43240c8a07e0bf019745d
Author: Paul Rogers 
Date:   2018-05-01T02:10:53Z

DRILL-6373: Fix nullable vector data vector type

Fixes the type of the data vector within a nullable vector. The data vector 
is Required (has no bits vector.) Accurate metadata is required for proper 
overflow handling in the result set loader.

commit 9496ef681f19f03ccd735f2c9b18f6d914eae3e2
Author: Paul Rogers 
Date:   2018-05-01T02:15:33Z

DRILL-6373: Refactor result set loader

commit 74675436ae1efdf66deafeaa27b281d169e274ad
Author: Paul Rogers 
Date:   2018-05-01T02:16:48Z

DRILL-6373: Revised projection & vector cache

commit 04598c0dbdbbff5ecbc2f89d02b14f66982f86bd
Author: Paul Rogers 
Date:   2018-05-01T02:17:22Z

DRILL-6373: Revised & added unit tests




> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.14.0
>
>
> As the next step in merging the 

[jira] [Comment Edited] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy

[Jiang Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459369#comment-16459369
 ] 

Jiang Wu edited comment on DRILL-6242 at 5/1/18 1:28 AM:
-

The resulting changes make use of java.time.Local as the 
Java object representation of values from respective Drill vector types: Date, 
Time, Timestamp.  With this change, accessing Drill date time vectors will 
return non time zone specific values.

Below lists out the behavior of Drill with examples to illustrate how the time 
values from various data sources (storage plugin, inline functions) are 
handled.  This represents existing behavior. 

For each data source, the example shows the original value in the data source, 
how such value is interpreted and converted into a value for the Drill 
Timestamp vector, how the value from the Timestamp vector is read.  And how the 
client of Drill can reproduce the original value from the LocalDateTime 
returned from the Drill Timestamp vector.  Any value that is different from the 
original value is highlighted in red.

A Timestamp represents an instant in time and in theory should not be timezone 
dependent.  We can interpret a Timestamp being made of 3 parts: date part, time 
part, and time zone/offset.  Based on the time zone/offset, the date part and 
the time part can be different for the same Timestamp instance.

*1. Date source: extended JSON file, TO_TIMESTAMP(), and CAST to TIMESTAMP.*  
Any time zone associated with the original time value is ignored.  This means 
all timestamps are treated as though they are from the Drill server's local 
time zone.  E.g.

 
{code:java}
select case when t1 = t2 then 1 else 0 end 
from (
  select TO_TIMESTAMP('2015-03-30 20:49:59.0 UTC', '-MM-dd HH:mm:ss.s z') 
as t1,
 TO_TIMESTAMP('2015-03-30 20:49:59.0 PST', '-MM-dd HH:mm:ss.s z') 
as t2
  from (values(1))
)
{code}
 

returns

 
{code:java}
+-+
| EXPR$0  |
+-+
| 1       |
+-+{code}
 

*2. Date source: parquet timestamp.* Treat date-part and time-part as though 
they are in the Drill server time zone.  Timestamp value is represented as a 
long in Parquet data source.  Produce a \{date, time, UTC} from the Timestamp, 
but then ignore the time zone.  The result is a data part and time part with 
the same values as seen from the UTC time zone.  Example:

 
|| ||Parquet Timestamp value||Write to Drill Timestamp Vector||Read from Drill 
Timestamp Vector||How to get back the original value||
|Actual 
value|123456789012|123456789012|{color:#ff}1973-11-29T21:33:09.012{color}
 
{color:#ff}(LocalDateTime){color}|"1973-11-29T21:33:09.012".atZone(OffsetZone.UTC).toInstant()|
|Interpretation in Drill Server Time 
Zone|1973-11-29T21:33:09Z|1973-11-29T21:33:09Z|{color:#ff}1973-11-29T21:33:09PST{color}|1973-11-29T21:33:09Z|

 

 

 
 *3. Date Source: parquet int96 as timestamp.* Generate the date-part and 
time-part in the Drill server time zone that represent the same instant as the 
timestamp.  Produce a \{date1, time1, UTC} from the Parquet int96 value.  
Convert this to another \{date2, time2, Drill Time Zone} representation.  Strip 
out the Drill Time Zone and replace with UTC resulting in a \{date2, time2, 
UTC} and store in vector.  Example:
  
|| ||Parquet int96 as Timestamp||Write to Drill Timestamp Vector||Read from 
Drill Timestamp Vector||How to get back the original value||
|Actual 
value|1312196153000|{color:#ff}1312170953000{color}|{color:#ff}2011-08-01T03:55:53{color}
 
{color:#ff}(LocalDateTime){color}|"2011-08-01T03:55:53".atZone(ZoneOffset.systemDefault()).toInstant()|
|Interpretation in Drill Server Time Zone|2011-08-01T10:55:53Z
  
 which is the same as:
  
 2011-08-01T03:55:53 
PDT|{color:#ff}2011-08-01T03:55:53Z{color}|{color:#ff}2011-08-01T03:55:53
 PDT{color}|2011-08-01T10:55:53Z|

*4. Date Source: BSON.* Same as Parquet int96 as timestamp type.  Preserve the 
correct time.  Produce a \{date1, time1, UTC} from the Parquet int96 value.  
Convert this to another \{date2, time2, Drill Time Zone} representation.  Strip 
out the Drill Time Zone and replace with UTC resulting in a \{date2, time2, 
UTC} and store in vector.  Example:
|| ||BSON DateTime (long)||Write to Drill Timestamp Vector||Read from Drill 
Timestamp Vector||How to get back the original value||
|Actual 
Value|5262729712|{color:#d04437}5233929712{color}|{color:#d04437}1970-03-02T13:52:09.712{color}
 
{color:#d04437}(LocalDateTime){color}|{color:#33}"1970-03-02T13:52:09.712".atZone(ZoneOffset.systemDefault()).toInstant(){color}|
|Interpretation in Drill Server Time Zone|1970-03-02T21:52:09Z
  
 which is the same as:
  
 1970-03-02T13:52:09 
PST|{color:#d04437}1970-03-02T13:52:09.712Z{color}|{color:#d04437}1970-03-02T13:52:09.712
 PST{color}
  |{color:#33}1970-03-02T21:52:09Z{color}|

 

 


was (Author: wu):
The resulting changes 

[jira] [Created] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support

[Paul Rogers (JIRA)]

Paul Rogers created DRILL-6373:
--

 Summary: Refactor the Result Set Loader to prepare for Union, List 
support
 Key: DRILL-6373
 URL: https://issues.apache.org/jira/browse/DRILL-6373
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.13.0
Reporter: Paul Rogers
Assignee: Paul Rogers
 Fix For: 1.14.0


As the next step in merging the "batch sizing" enhancements, refactor the 
{{ResultSetLoader}} and related classes to prepare for Union and List support. 
This fix follows the refactoring of the column accessors for the same purpose. 
Actual Union and List support is to follow in a separate PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy

[Jiang Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459369#comment-16459369
 ] 

Jiang Wu commented on DRILL-6242:
-

The resulting changes make use of java.time.Local as the 
Java object representation of values from respective Drill vector types: Date, 
Time, Timestamp.  With this change, accessing Drill date time vectors will 
return non time zone specific values.

Below lists out the behavior of Drill with examples to illustrate how the time 
values from various data sources (storage plugin, inline functions) are 
handled.  This represents existing behavior. 

For each data source, the example shows the original value in the data source, 
how such value is interpreted and converted into a value for the Drill 
Timestamp vector, how the value from the Timestamp vector is read.  And how the 
client of Drill can reproduce the original value from the LocalDateTime 
returned from the Drill Timestamp vector.  Any value that is different from the 
original value is highlighted in red.

A Timestamp represents an instant in time and in theory should not be timezone 
dependent.  We can interpret a Timestamp being made of 3 parts: date part, time 
part, and time zone/offset.  Based on the time zone/offset, the date part and 
the time part can be different for the same Timestamp instance.

*1. Date source: extended JSON file, TO_TIMESTAMP(), and CAST to TIMESTAMP.*  
Any time zone associated with the original time value is ignored.  This means 
all timestamps are treated as though they are from the Drill server's local 
time zone.  E.g.

 
{code:java}
select case when t1 = t2 then 1 else 0 end 
from (
  select TO_TIMESTAMP('2015-03-30 20:49:59.0 UTC', '-MM-dd HH:mm:ss.s z') 
as t1,
 TO_TIMESTAMP('2015-03-30 20:49:59.0 PST', '-MM-dd HH:mm:ss.s z') 
as t2
  from (values(1))
)
{code}
 

returns

 
{code:java}
+-+
| EXPR$0  |
+-+
| 1       |
+-+{code}
 

*2. Date source: parquet timestamp.* Treat date-part and time-part as though 
they are in the Drill server time zone.  Timestamp value is represented as a 
long in Parquet data source.  Produce a \{date, time, UTC} from the Timestamp, 
but then ignore the time zone.  The result is a data part and time part with 
the same values as seen from the UTC time zone.  Example:

 
|| ||Parquet Timestamp value||Write to Drill Timestamp Vector||Read from Drill 
Timestamp Vector||How to get back the original value||
|Actual 
value|123456789012|123456789012|{color:#FF}1973-11-29T21:33:09.012{color}
{color:#FF}(LocalDateTime){color}|"1973-11-29T21:33:09.012".atZone(OffsetZone.UTC).toInstant()|
|Interpretation in Drill Server Time 
Zone|1973-11-29T21:33:09Z|1973-11-29T21:33:09Z|{color:#FF}1973-11-29T21:33:09PST{color}|1973-11-29T21:33:09Z|

 

 

 
*3. Date Source: parquet int96 as timestamp.* Generate the date-part and 
time-part in the Drill server time zone that represent the same instant as the 
timestamp.  Produce a \{date1, time1, UTC} from the Parquet int96 value.  
Convert this to another \{date2, time2, Drill Time Zone} representation.  Strip 
out the Drill Time Zone and replace with UTC resulting in a \{date2, time2, 
UTC} and store in vector.  Example:
 
|| ||Parquet int96 as Timestamp||Write to Drill Timestamp Vector||Read from 
Drill Timestamp Vector||How to get back the original value||
|Actual 
value|1312196153000|{color:#FF}1312170953000{color}|{color:#FF}2011-08-01T03:55:53{color}
{color:#FF}(LocalDateTime){color}|"2011-08-01T03:55:53".atZone(ZoneOffset.systemDefault()).toInstant()|
|Interpretation in Drill Server Time Zone|2011-08-01T10:55:53Z
 
which is the same as:
 
2011-08-01T03:55:53 
PDT|{color:#FF}2011-08-01T03:55:53Z{color}|{color:#FF}2011-08-01T03:55:53
 PDT{color}|2011-08-01T10:55:53Z|

*4. Date Source: BSON.* Same as Parquet int96 as timestamp type.  Preserve the 
correct time.  Produce a \{date1, time1, UTC} from the Parquet int96 value.  
Convert this to another \{date2, time2, Drill Time Zone} representation.  Strip 
out the Drill Time Zone and replace with UTC resulting in a \{date2, time2, 
UTC} and store in vector.  Example:
|| ||BSON DateTime (long)||Write to Drill Timestamp Vector||Read from Drill 
Timestamp Vector||How to get back the original value||
|Actual 
Value|5262729712|{color:#d04437}5233929712{color}|{color:#d04437}1970-03-02T13:52:09.712{color}
{color:#d04437}(LocalDateTime){color}|"1970-03-02T13:52:09.712".atZone(ZoneOffset.systemDefault()).toInstant()|
|Interpretation in Drill Server Time Zone|1970-03-02T21:52:09Z
 
which is the same as:
 
1970-03-02T13:52:09 
PST|{color:#d04437}1970-03-02T13:52:09.712Z{color}|{color:#d04437}1970-03-02T13:52:09.712
 PST{color}
 
|1970-03-02T21:52:09Z|

 

 

> Output format for nested date, time, timestamp values in an object hierarchy
> 

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459245#comment-16459245
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185135884
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -103,14 +105,14 @@
 
 ${drillbit.getState()}
 Not Available
-  
-<#if ( model.shouldShowAdminInfo() &&  ( 
drillbit.isCurrent() || ( !model.isAuthEnabled() && location.protocol != 
"https" ))) >
-  
-<#else>
+
+  <#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() || drillbit.isCurrent() ) >
--- End diff --

Yes, and (unfortunately), I can't check for 'https' protocol within 
freemarker. The check for this is taken care of at [line 
317](https://github.com/kkhatua/drill/blob/ab3e8619c6259803eb362be290a3a3605839a194/exec/java-exec/src/main/resources/rest/index.ftl#L317)
 of `updateStatusAndShutdown` function, where I check for *HTTPS* and as a 
current Drillbit.
```javascript
if (status_map[key] == "ONLINE") {
$("#row-"+i).find("#status").text(status_map[key]);
<#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() ) >
if ( location.protocol != "https" || 
($("#row-"+i).find("#current").html() == "Current") ) {
  
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0).css('cursor','pointer');
}

```



> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459232#comment-16459232
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185134711
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -328,9 +432,11 @@
   //Iterates through all the nodes for update
   function reloadMetrics() {
   for (i = 1; i <= size; i++) {
-  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
-  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
-  updateMetricsHtml(address, httpPort, i);
+  if ( $("#row-"+i).find("#stateKey").length == 0 ) {
--- End diff --

Very odd. The `stateKey` seems to disappear (or never appeared). Let me 
look at what happened.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459214#comment-16459214
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185131588
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -328,9 +432,11 @@
   //Iterates through all the nodes for update
   function reloadMetrics() {
   for (i = 1; i <= size; i++) {
-  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
-  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
-  updateMetricsHtml(address, httpPort, i);
+  if ( $("#row-"+i).find("#stateKey").length == 0 ) {
+var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+updateMetricsHtml(address, httpPort, i);
--- End diff --

Ok. Logically, this seems to resolve to the same outcome, though.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459210#comment-16459210
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185131321
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
+  key = $("#row-"+i).find("#stateKey").textContent;
+} else {
+  let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  let port = $("#row-"+i).find("#httpPort").html();
+  key = address+"-"+port;
+}
 
-if (status_map[key] == null) {
+if (typeof status_map == 'undefined') {
+$("#row-"+i).find("#status").text(nAText);
+
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
+$("#row-"+i).find("#queriesCount").text("");
+} else if (status_map[key] == null) {
 $("#row-"+i).find("#status").text("OFFLINE");
 
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
 $("#row-"+i).find("#queriesCount").text("");
-}
-else {
+} else {
 if (status_map[key] == "ONLINE") {
 $("#row-"+i).find("#status").text(status_map[key]);
-
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0);
-}
-else {
+<#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() ) >
+if ( location.protocol != "https" || 
($("#row-"+i).find("#current").html() == "Current") ) {
+  
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0).css('cursor','pointer');
+}
+
+} else {
 if ($("#row-"+i).find("#current").html() == "Current") 
{
 fillQueryCount(i);
 }
 $("#row-"+i).find("#status").text(status_map[key]);
 }
+//Removing accounted key
+delete bitMap[key];
  

[jira] [Resolved] (DRILL-5968) C++ Client should set service_host to the connected host if service_host is empty

[Parth Chandra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Chandra resolved DRILL-5968.
--
Resolution: Fixed

> C++ Client should set service_host to the connected host if service_host is 
> empty
> -
>
> Key: DRILL-5968
> URL: https://issues.apache.org/jira/browse/DRILL-5968
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Parth Chandra
>Priority: Major
>
> In the ODBC driver - 
> The krbSpnConfigurationsRequired parameter in odbc.ini is not working as 
> expected.  If I set the following:
> AuthenticationType=Kerberos
> UID=
> PWD=
> DelegationUID=
> KrbServiceName=
> KrbServiceHost=maprsasl
> krbSpnConfigurationsRequired=1
> I could connect successfully.  I was expecting to get an error message.  If 
> only either KrbServiceHost is missing or both KrbServiceHost and 
> KrbServiceName are missing then I get the expected error message.
> Turning off the parameter, I was able to connect using the following setting:
> AuthenticationType=Kerberos
> UID=
> PWD=
> DelegationUID=
> KrbServiceName=
> KrbServiceHost=maprsasl
> krbSpnConfigurationsRequired=0
> However, if either KrbServiceHost or both KrbServiceHost and KrbServiceName 
> are missing, I would get the following error message:
> 1: SQLDriverConnect = [MapR][Drill] (30)  User authentication failed. Server 
> message: DrillClientImpl::handleAuthentication: Authentication failed. 
> [Details:  Encryption: enabled ,MaxWrappedSize: 32768 ,WrapSizeLimit: 0, 
> Error: -1]. Check connection parameters? (30) SQLSTATE=28000
> 1: ODBC_Connect = [MapR][Drill] (30)  User authentication failed. Server 
> message: DrillClientImpl::handleAuthentication: Authentication failed. 
> [Details:  Encryption: enabled ,MaxWrappedSize: 32768 ,WrapSizeLimit: 0, 
> Error: -1]. Check connection parameters? (30) SQLSTATE=28000
> The Drill C++ Client should set service_host to the connected host (if 
> direct) if service_host is empty (similar logic for zookeeper connection). 
> Going through the source code, looks like this logic was removed by this 
> commit: 
> https://github.com/apache/drill/commit/f246c3cad7f44baeb8153913052ebc963c62276a#diff-8e6df071d8ca863fcfa578892944c1dcL123



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459201#comment-16459201
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185129153
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
+  key = $("#row-"+i).find("#stateKey").textContent;
+} else {
+  let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  let port = $("#row-"+i).find("#httpPort").html();
+  key = address+"-"+port;
+}
 
-if (status_map[key] == null) {
+if (typeof status_map == 'undefined') {
+$("#row-"+i).find("#status").text(nAText);
+
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
+$("#row-"+i).find("#queriesCount").text("");
+} else if (status_map[key] == null) {
 $("#row-"+i).find("#status").text("OFFLINE");
 
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
 $("#row-"+i).find("#queriesCount").text("");
-}
-else {
+} else {
 if (status_map[key] == "ONLINE") {
 $("#row-"+i).find("#status").text(status_map[key]);
-
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0);
-}
-else {
+<#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() ) >
+if ( location.protocol != "https" || 
($("#row-"+i).find("#current").html() == "Current") ) {
+  
$("#row-"+i).find("#shutdown").prop('disabled',false).css('opacity',1.0).css('cursor','pointer');
+}
+
+} else {
 if ($("#row-"+i).find("#current").html() == "Current") 
{
 fillQueryCount(i);
 }
 $("#row-"+i).find("#status").text(status_map[key]);
 }
+//Removing accounted key
+delete bitMap[key];
  

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459199#comment-16459199
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185128740
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
--- End diff --

`bit_map` is a local map which is originally a clone of `status_map` before 
we remove already-seen bits. The remainder of the map is passed on. 
`status_map` is a global variable, as it is shared by multiple AJAX calls, so 
we avoid using that in `listNewDrillbits`


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459176#comment-16459176
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185127293
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
--- End diff --

I'll check this and get back to you.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459167#comment-16459167
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185126155
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
--- End diff --

Ok. A lot of functions need comments. I'll update even those that haven't 
been touched.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459165#comment-16459165
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185125707
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
--- End diff --

Unused variable name required for jQuery. I'll fix this with a better name 
:)


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6236) batch sizing for hash join

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6236:
-
Reviewer: Boaz Ben-Zvi

> batch sizing for hash join
> --
>
> Key: DRILL-6236
> URL: https://issues.apache.org/jira/browse/DRILL-6236
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Major
> Fix For: 1.14.0
>
>
> limit output batch size for hash join based on memory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-6351) Drill fails with NullPointerException when starting in embedded mode

[Volodymyr Vysotskyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Volodymyr Vysotskyi resolved DRILL-6351.

Resolution: Fixed

Fixed in DRILL-6302

> Drill fails with NullPointerException when starting in embedded mode
> 
>
> Key: DRILL-6351
> URL: https://issues.apache.org/jira/browse/DRILL-6351
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Volodymyr Vysotskyi
>Assignee: Kunal Khatua
>Priority: Blocker
> Fix For: 1.14.0
>
>
> When starting Drill in embedded mode and another Drill instance is already 
> running, it fails with NPE:
> {noformat}
> java.lang.NullPointerException
> at 
> org.apache.drill.exec.coord.local.LocalClusterCoordinator.update(LocalClusterCoordinator.java:98)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:231)
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.cleanup(DrillConnectionImpl.java:827)
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:186)
> at 
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at 
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:68)
> at 
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at 
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.close(Commands.java:925)
> at sqlline.Commands.quit(Commands.java:889)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.begin(SqlLine.java:621)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> {noformat}
> Before the exception was:
> {noformat}
> Error: Failure in starting embedded Drillbit: java.net.BindException: Address 
> already in use (state=,code=0)
> java.sql.SQLException: Failure in starting embedded Drillbit: 
> java.net.BindException: Address already in use
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:144)
> at 
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at 
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:68)
> at 
> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at sqlline.DatabaseConnection.connect(DatabaseConnection.java:167)
> at 
> sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:213)
> at sqlline.Commands.connect(Commands.java:1083)
> at sqlline.Commands.connect(Commands.java:1015)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:742)
> at sqlline.SqlLine.initArgs(SqlLine.java:528)
> at sqlline.SqlLine.begin(SqlLine.java:596)
> at sqlline.SqlLine.start(SqlLine.java:375)
> at sqlline.SqlLine.main(SqlLine.java:268)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at 
> org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:279)
> at 
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
> at 
> org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:218)
> at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>   

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459085#comment-16459085
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185064331
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
--- End diff --

You can get rid of this else and just call `updateStatusAndShutdown` 
instead. That way inside the `.done callback of .getJson `also you don't have 
to call `updateStatusAndShutdown` explicitly.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459083#comment-16459083
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185094337
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
+  key = $("#row-"+i).find("#stateKey").textContent;
+} else {
+  let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  let port = $("#row-"+i).find("#httpPort").html();
+  key = address+"-"+port;
+}
 
-if (status_map[key] == null) {
+if (typeof status_map == 'undefined') {
+$("#row-"+i).find("#status").text(nAText);
+
$("#row-"+i).find("#shutdown").prop('disabled',true).css('opacity',0.5);
+$("#row-"+i).find("#queriesCount").text("");
+} else if (status_map[key] == null) {
--- End diff --

There is a different mode called `Quiescent` where Drillbit will be OFFLINE 
but will not deregister itself from Zookeeper. In that case showing Drillbit as 
`OFFLINE` is fine as compared to when Drillbit has deregistered itself from 
Zookeeper. 
`status_map[key] `will be `null` for second scenario where I guess we 
should not show any info as it's not available.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if 

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459091#comment-16459091
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185066896
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
--- End diff --

Please store the reference to the row object and use it here and below 
rather than searching it every time. 
`let currentRow = $("#row-"+i)`
Also comment looks wrong since `if condition `is checking for if there is 
any row with stateKey instead ?


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459086#comment-16459086
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185072034
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -252,33 +255,129 @@
   timeout = setTimeout(reloadStatus, refreshTime);
   }
 
-  function fillStatus(data,size) {
-  var status_map = (data.responseJSON);
-  for (i = 1; i <= size; i++) {
-var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue;
-address = address.trim();
-var port = $("#row-"+i).find("#port").html();
-var key = address+"-"+port;
+  function fillStatus(dataResponse,size) {
+  var status_map = (dataResponse.responseJSON);
+  //In case localhost has gone down (i.e. we don't know status 
from ZK)
+  if (typeof status_map == 'undefined') {
+//Query other nodes for state details
+for (j = 1; j <= size; j++) {
+  if ($("#row-"+j).find("#current").html() == "Current") {
+continue; //Skip LocalHost
+  }
+  var address = 
$("#row-"+j).find("#address").contents().get(0).nodeValue.trim();
+  var restPort = 
$("#row-"+j).find("#httpPort").contents().get(0).nodeValue.trim();
+  var altStateUrl = location.protocol + "//" + 
address+":"+restPort + "/state";
+  var goatResponse = $.getJSON(altStateUrl)
+.done(function(stateDataJson) {
+//Update Status & Buttons for alternate stateData
+if (typeof status_map == 'undefined') {
+  status_map = (stateDataJson); //Update
+  updateStatusAndShutdown(stateDataJson);
+}
+  });
+  //Don't loop any more
+  if (typeof status_map != 'undefined') {
+break;
+  }
+}
+  } else {
+updateStatusAndShutdown(status_map);
+  }
+  }
+
+  function updateStatusAndShutdown(status_map) {
+let bitMap = {};
+if (typeof status_map != 'undefined') {
+for (var k in status_map) {
+  bitMap[k] = status_map[k];
+}
+}
+for (i = 1; i <= size; i++) {
+let key = "";
+if ($("#row-"+i).find("#stateKey").length > 0) { //Check if 
newBit that has no stateKey
+  key = $("#row-"+i).find("#stateKey").textContent;
+} else {
+  let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+  let port = $("#row-"+i).find("#httpPort").html();
+  key = address+"-"+port;
+}
--- End diff --

the if-else condition can be removed. I don't think stateKey is needed at 
all.

```
let address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
let port = $("#row-"+i).find("#httpPort").html();
key = address+"-"+port;
```


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459087#comment-16459087
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185113634
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -328,9 +432,11 @@
   //Iterates through all the nodes for update
   function reloadMetrics() {
   for (i = 1; i <= size; i++) {
-  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
-  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
-  updateMetricsHtml(address, httpPort, i);
+  if ( $("#row-"+i).find("#stateKey").length == 0 ) {
+var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
+updateMetricsHtml(address, httpPort, i);
--- End diff --

Please update the check inside `updateMetricsHtml`, it will not refresh for 
local bit in case when Auth is enabled.
```
Current:
if ( !updateRemoteInfo || (location.protocol == "https" && remoteHost != 
location.host) ) {
   return;
}

Should Be:
if ( (!updateRemoteInfo || (location.protocol == "https") && remoteHost != 
location.host ) {
   return;
}
```


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459088#comment-16459088
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185114420
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -328,9 +432,11 @@
   //Iterates through all the nodes for update
   function reloadMetrics() {
   for (i = 1; i <= size; i++) {
-  var address = 
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
-  var httpPort = 
$("#row-"+i).find("#httpPort").contents().get(0).nodeValue.trim();
-  updateMetricsHtml(address, httpPort, i);
+  if ( $("#row-"+i).find("#stateKey").length == 0 ) {
--- End diff --

This check means we are not loading the metrics for newly available 
Drillbits, why is that ? 

I think instead we should check if the `status is ONLINE` then only 
`updateMetricsHtml`.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6364) WebUI does not cleanly handle shutdown and state toggling when Drillbits go on and offline

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459090#comment-16459090
 ] 

ASF GitHub Bot commented on DRILL-6364:
---

Github user sohami commented on a diff in the pull request:

https://github.com/apache/drill/pull/1241#discussion_r185115283
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -103,14 +105,14 @@
 
 ${drillbit.getState()}
 Not Available
-  
-<#if ( model.shouldShowAdminInfo() &&  ( 
drillbit.isCurrent() || ( !model.isAuthEnabled() && location.protocol != 
"https" ))) >
-  
-<#else>
+
+  <#if ( model.shouldShowAdminInfo() || 
!model.isAuthEnabled() || drillbit.isCurrent() ) >
--- End diff --

This check doesn't look correct to me. `shouldShowAdminInfo` checks If the 
logged in user is admin or not. So for admin user it will show the shutdown 
button in if block even for remote Drillbit which is not the intention.


> WebUI does not cleanly handle shutdown and state toggling when Drillbits go 
> on and offline
> --
>
> Key: DRILL-6364
> URL: https://issues.apache.org/jira/browse/DRILL-6364
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Web Server
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When the webpage is loaded the first time, the shutdown button is enabled by 
> default, which might not be correct, since scenarios like HTTPS, etc does not 
> support this for remote bits. (i.e the user needs to navigate to that node's 
> UI for shutting it down). 
> Similarly, when a previously unseen Drillbit comes online, the node will not 
> be rendered until the page is refreshed by the user. 
> Lastly, if the node from whom the UI page was served goes down, the status 
> update for the rest of the cluster is not updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458994#comment-16458994
 ] 

ASF GitHub Bot commented on DRILL-6321:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/1224
  
@vvysotskyi  The lateral join implementation committed in DRILL-6323 
supports cross join for all conditions. The main difficulty in cross join is 
the management of the memory needed to produce a cartesian product (M x N). In 
cross apply though, the cross join is only computed on one row in the left side 
at a time (1 x N), since the left and right sides are correlated. This allows 
cross apply to operate with a very small amount of memory. 


> Lateral Join: Planning changes - enable submitting physical plan
> 
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
>  Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
>
> Implement changes to enable submitting a physical plan containing lateral and 
> unnest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-143) Support CGROUPs resource management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458877#comment-16458877
 ] 

ASF GitHub Bot commented on DRILL-143:
--

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1239
  
Yes, I agree. Support folks gave me similar feedback, so I'll commit this 
change in the mapr distro _IFF_ there is a request for that. YARN is already a 
complex beast with numerous settings. Introducing caveats like this will only 
add to the confusion.


> Support CGROUPs resource management
> ---
>
> Key: DRILL-143
> URL: https://issues.apache.org/jira/browse/DRILL-143
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.14.0
>
> Attachments: 253ce178-ddeb-e482-cd64-44ab7284ad1c.sys.drill
>
>
> For the purpose of playing nice on clusters that don't have YARN, we should 
> write up configuration and scripts to allows users to run Drill next to 
> existing workloads without sharing resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-143) Support CGROUPs resource management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458866#comment-16458866
 ] 

ASF GitHub Bot commented on DRILL-143:
--

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/1239
  
Just to be clear, I have no objection Drill enforcing its own cgroup limits.

My point is rather that CPU limits must be integrated with YARN, via the 
DoY config file, so that the user specifies CPU limits in one place and that 
limit is the same one passed to YARN for container allocation and to whichever 
code is doing cgroup enforcement. By default, it will be YARN if cgroups are 
enabled in YARN, as they can be for Apache YARN.

It is then fine to have an additional option to enable self-enforcement in 
Drill for those odd cases where users cannot or choose not to use versions of 
YARN that do the work.

The problem would be if the user has to configure CPU in two places: in a 
shell script to enable self-enforcement, and in the DoY config to tell YARN the 
container size. This goes against the ease-of-use experience DoY was designed 
to provide. (That is, getting multiple to work consistently correct is very 
hard if you're just trying to get Drill to run and are not a Drill internals 
expert.)


> Support CGROUPs resource management
> ---
>
> Key: DRILL-143
> URL: https://issues.apache.org/jira/browse/DRILL-143
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.14.0
>
> Attachments: 253ce178-ddeb-e482-cd64-44ab7284ad1c.sys.drill
>
>
> For the purpose of playing nice on clusters that don't have YARN, we should 
> write up configuration and scripts to allows users to run Drill next to 
> existing workloads without sharing resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458857#comment-16458857
 ] 

ASF GitHub Bot commented on DRILL-6348:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1237#discussion_r185063074
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java 
---
@@ -77,4 +83,47 @@ public long getByteCount() {
   public boolean isAckSent() {
 return ackSent.get();
   }
+
+  /**
+   * Transfer ownership of this DrillBuf to the target allocator. This is 
done for better memory
+   * accounting (that is, the operator should be charged with the body's 
Drillbuf memory).
+   *
+   * NOTES -
+   * 
+   * This operation is a NOOP when a) the current allocator 
(associated with the DrillBuf) is not the
+   * owning allocator or b) the target allocator is already the owner
+   * When transfer happens, a new RawFragmentBatch instance is 
allocated; this is done for proper
+   * DrillBuf reference count accounting
+   * The RPC handling code caches a reference to this RawFragmentBatch 
object instance; release()
+   * calls should be routed to the previous DrillBuf
+   * Caller is responsible for checking an OOM condition
+   * 
+   *
+   * @param targetAllocator target allocator
+   * @return a new {@link RawFragmentBatch} object instance on success 
(where the buffer ownership has
+   * been switched to the target allocator); otherwise this 
operation is a NOOP (current instance
+   * returned)
+   */
+  public RawFragmentBatch transferBodyOwnership(BufferAllocator 
targetAllocator) {
--- End diff --

DrillBuf has a transferOwnership method that could be used instead? See the 
transferTo methods in any of the vector classes -  
[FixedValueVectors.transferTo()](https://github.com/apache/drill/blob/master/exec/vector/src/main/codegen/templates/FixedValueVectors.java#L282)
 
Saves you the trouble of rolling your own and also avoids the need to make 
changes to any of the existing classes.


> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
>  Issue Type: Task
>  Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the 
> Unordered Receiver operator. This is problematic when analyzing OOM 
> conditions since we cannot account for all of a query memory usage. This Jira 
> is to fix memory reporting for the Unordered Receiver operator.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6348) Unordered Receiver does not report its memory usage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458856#comment-16458856
 ] 

ASF GitHub Bot commented on DRILL-6348:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1237#discussion_r185064842
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java
 ---
@@ -182,13 +189,15 @@ public IterOutcome next() {
 return IterOutcome.OUT_OF_MEMORY;
   }
 
+  // Transfer the ownership of this raw-batch to this operator for 
proper memory statistics reporting
+  batch = batch.transferBodyOwnership(oContext.getAllocator());
--- End diff --

This should probably be done inside the [ RecordBatchLoader.load() 
](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/record/RecordBatchLoader.java#L78)
 method.
Note that `MergingRecordBatch` has similar code and so probably suffers 
from the same memory accounting issue. All other uses of 
`RecordBatchLoader.load()` appear to be in the client code or test code so we 
are unlikely to break anything by making the change in `RecordBatchLoader`.



> Unordered Receiver does not report its memory usage
> ---
>
> Key: DRILL-6348
> URL: https://issues.apache.org/jira/browse/DRILL-6348
> Project: Apache Drill
>  Issue Type: Task
>  Components: Execution - Flow
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Drill Profile functionality doesn't show any memory usage for the 
> Unordered Receiver operator. This is problematic when analyzing OOM 
> conditions since we cannot account for all of a query memory usage. This Jira 
> is to fix memory reporting for the Unordered Receiver operator.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-143) Support CGROUPs resource management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458850#comment-16458850
 ] 

ASF GitHub Bot commented on DRILL-143:
--

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1239
  
@paul-rogers I went through with support on this and found that the issue 
is not specific to MapR. However, you make a strong argument in favor of 
letting YARN handle the CGroup management rather than Drill over-reaching. 
I've reverted the change to `yarn-drillbit.sh` in the latest commit.


> Support CGROUPs resource management
> ---
>
> Key: DRILL-143
> URL: https://issues.apache.org/jira/browse/DRILL-143
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.14.0
>
> Attachments: 253ce178-ddeb-e482-cd64-44ab7284ad1c.sys.drill
>
>
> For the purpose of playing nice on clusters that don't have YARN, we should 
> write up configuration and scripts to allows users to run Drill next to 
> existing workloads without sharing resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5261) Expose REST endpoint in zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458837#comment-16458837
 ] 

ASF GitHub Bot commented on DRILL-5261:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1042
  
@xhochy I recently had a PR #1203  for DRILL-6289 and realized your PR's 
solution was a subset of that. I've already incorporated changes to the C++ 
client's protobuf as well. If it addresses the requirements you had for this 
JIRA, can you close the PR? If there is something missing, feel free to submit 
a new PR with the missing feature.


> Expose REST endpoint in zookeeper
> -
>
> Key: DRILL-5261
> URL: https://issues.apache.org/jira/browse/DRILL-5261
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Uwe L. Korn
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> It would be nice to also publish the REST API endpoint of each Drillbit in 
> the Zookeeper. This would mean that we need an additional entry in 
> {{DrillbitEndpoint}}. While I would know how to add the attribute to the 
> ProtoBuf definition and filling the attribute with the correct information, 
> I'm unsure if there is the need for some migration code to support older 
> {{DrillbitEndpoint}} implementations that don't have this attribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6372) SQL Queries fails with DB2

[Juha (JIRA)]

Juha created DRILL-6372:
---

 Summary: SQL Queries fails with DB2
 Key: DRILL-6372
 URL: https://issues.apache.org/jira/browse/DRILL-6372
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - Java
Affects Versions: 1.3.0
 Environment: Centos 7.4.1 and Mac OS-X (10.13.4) . Java 1.8 build 
1.8.0_102-b14 . DB2 11.1.0

 
Reporter: Juha


Queries like _SELECT 'test' FROM db2.SYSIBM.SYSDUMMY1_ fails with the lates db2 
driver 4.21.29 (More: 
https://stackoverflow.com/questions/49710342/error-when-using-db2-from-apache-drill)

Connection:

{
 "type": "jdbc",
 "driver": "com.ibm.db2.jcc.DB2Driver",
 "url": "jdbc:db2://host:5/TESTDB",
 "username": "db2inst1",
 "password": "XXX",
 "enabled": true
}

Query throws following stack trace: 

at 
org.apache.drill.exec.store.jdbc.JdbcStoragePlugin$JdbcCatalogSchema.setHolder(JdbcStoragePlugin.java:346)
 ~[drill-jdbc-storage-1.13.0.jar:1.13.0]
 at 
org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.registerSchemas(JdbcStoragePlugin.java:434)
 ~[drill-jdbc-storage-1.13.0.jar:1.13.0]
 at 
org.apache.calcite.jdbc.DynamicRootSchema.loadSchemaFactory(DynamicRootSchema.java:81)
 ~[drill-java-exec-1.13.0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.jdbc.DynamicRootSchema.getImplicitSubSchema(DynamicRootSchema.java:66)
 ~[drill-java-exec-1.13.0.jar:1.15.0-drill-r0]
 at org.apache.calcite.jdbc.CalciteSchema.getSubSchema(CalciteSchema.java:233) 
~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SqlValidatorUtil.getSchema(SqlValidatorUtil.java:992)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SqlValidatorUtil.getTableEntry(SqlValidatorUtil.java:953)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:117)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.drill.exec.planner.sql.SqlConverter$DrillCalciteCatalogReader.getTable(SqlConverter.java:633)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
org.apache.drill.exec.planner.sql.SqlConverter$DrillValidator.validateFrom(SqlConverter.java:261)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3216)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:947)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:928)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:226) 
~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:903)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:613)
 ~[calcite-core-1.15.0-drill-r0.jar:1.15.0-drill-r0]
 at 
org.apache.drill.exec.planner.sql.SqlConverter.validate(SqlConverter.java:190) 
[drill-java-exec-1.13.0.jar:1.13.0]
 ... 10 common frames omitted
2018-04-14 14:27:08,776 [qtp2102527385-110] ERROR 
o.a.d.e.server.rest.QueryResources - Query from Web UI Failed
org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: null

SQL Query null

[Error Id: 16121ad3-919b-44cb-b343-b71ec56314f7 on 10.21.238.244:31010]
 at 
org.apache.drill.exec.rpc.AbstractDisposableUserClientConnection.sendResult(AbstractDisposableUserClientConnection.java:85)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:782)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
org.apache.drill.exec.work.foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:325)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
org.apache.drill.exec.work.foreman.QueryStateProcessor.planning(QueryStateProcessor.java:221)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83)
 ~[drill-java-exec-1.13.0.jar:1.13.0]
 at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:281) 
~[drill-java-exec-1.13.0.jar:1.13.0]
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[na:1.8.0_144]
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[na:1.8.0_144]
 at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
2018-04-14 

[jira] [Updated] (DRILL-3855) Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-3855:
---
Description: 
Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
reported in DRILL-3257, FilterSetOpTransposeRule, 
DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
Calcite, we have to enable these two rules to lift the performance. 

In addition, will update the plan validation in Unit test in response of the 
newly enabled rules. 

To cover complex cases (when some non project or filter operator is placed 
under the LogicalUnion operator) 
these rules should be added to main logical stage for Volcano Planner - 
DRILL-6371.

  was:
Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
reported in DRILL-3257, FilterSetOpTransposeRule, 
DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
Calcite, we have to enable these two rules to lift the performance. 

In addition, will update the plan validation in Unit test in response of the 
newly enabled rules. 

To cover complex cases (with more than two subqueries) these rules should be 
added to main logical stage for Volcano Planner - DRILL-6371.


> Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule
> ---
>
> Key: DRILL-3855
> URL: https://issues.apache.org/jira/browse/DRILL-3855
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects Versions: 1.13.0
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Vitalii Diravka
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
> reported in DRILL-3257, FilterSetOpTransposeRule, 
> DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
> Calcite, we have to enable these two rules to lift the performance. 
> In addition, will update the plan validation in Unit test in response of the 
> newly enabled rules. 
> To cover complex cases (when some non project or filter operator is placed 
> under the LogicalUnion operator) 
> these rules should be added to main logical stage for Volcano Planner - 
> DRILL-6371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6371) Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical stage

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-6371:
---
Description: 
FilterSetOpTransposeRule, ProjectSetOpTransposeRule are leveraged in DRILL-3855.
They are used in HepPlanner, but if they additionally will be enabled in main 
logical planning stage for Volcano planner, more cases will be covered with 
these rules.
For example: 
{code}
WITH year_total_1
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c),
 year_total_2
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c)
SELECT count(t_w_firstyear.customer_id) as ct
FROM   year_total_1 t_w_firstyear,
   year_total_2 t_w_secyear
WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
 AND t_w_firstyear.year_total > 0 and t_w_secyear.year_total > 0
{code}

Current plan after performing rules:
{code}
LogicalAggregate(group=[{}], ct=[COUNT($0)])
  LogicalProject(customer_id=[$0])
LogicalFilter(condition=[AND(=($1, $3), >($1, 0), >($3, 0))])
  LogicalJoin(condition=[true], joinType=[inner])
LogicalUnion(all=[true])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/region.parquet]])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
LogicalUnion(all=[true])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/region.parquet]])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
{code}
Since LogicalFilter isn't under LogicalUnion the FilterSetOpTransposeRule is 
not performed. 
FilterJoinRule from main Drill logical stage pushes LogicalFilter below, but 
the stage with FilterSetOpTransposeRule is already finished. 
That's why FilterSetOpTransposeRule and ProjectSetOpTransposeRule should be 
used in Drill main logical stage with Volcano planner.

Currently using them in Volcano Planner can cause infinite loops - CALCITE-1271

  was:
FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule are leveraged in 
DRILL-3855.
They are used in HepPlanner, but if they additionally will be enabled in main 
logical planning stage for Volcano planner, more cases will be covered with 
these rules.
For example: 
{code}
WITH year_total_1
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c),
 year_total_2
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c)
SELECT count(t_w_firstyear.customer_id) as ct
FROM   year_total_1 t_w_firstyear,
   year_total_2 t_w_secyear
WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
 AND t_w_firstyear.year_total > 0 and t_w_secyear.year_total > 0
{code}

Current plan after performing rules:
{code}
LogicalAggregate(group=[{}], ct=[COUNT($0)])
  LogicalProject(customer_id=[$0])
LogicalFilter(condition=[AND(=($1, $3), >($1, 0), >($3, 0))])
  LogicalJoin(condition=[true], joinType=[inner])
LogicalUnion(all=[true])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/region.parquet]])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
LogicalUnion(all=[true])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/region.parquet]])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
{code}
Since LogicalFilter isn't under LogicalUnion the 

Currently using them in Volcano Planner can cause infinite loops - CALCITE-1271


> Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical 
> stage
> --
>
> Key: DRILL-6371
> URL: https://issues.apache.org/jira/browse/DRILL-6371
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects 

[jira] [Updated] (DRILL-6371) Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical stage

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-6371:
---
Description: 
FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule are leveraged in 
DRILL-3855.
They are used in HepPlanner, but if they additionally will be enabled in main 
logical planning stage for Volcano planner, more cases will be covered with 
these rules.
For example: 
{code}
WITH year_total_1
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c),
 year_total_2
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c)
SELECT count(t_w_firstyear.customer_id) as ct
FROM   year_total_1 t_w_firstyear,
   year_total_2 t_w_secyear
WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
 AND t_w_firstyear.year_total > 0 and t_w_secyear.year_total > 0
{code}

Current plan after performing rules:
{code}
LogicalAggregate(group=[{}], ct=[COUNT($0)])
  LogicalProject(customer_id=[$0])
LogicalFilter(condition=[AND(=($1, $3), >($1, 0), >($3, 0))])
  LogicalJoin(condition=[true], joinType=[inner])
LogicalUnion(all=[true])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/region.parquet]])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
LogicalUnion(all=[true])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/region.parquet]])
  LogicalProject(customer_id=[$1], year_total=[1])
EnumerableTableScan(table=[[cp, tpch/nation.parquet]])
{code}
Since LogicalFilter isn't under LogicalUnion the 

Currently using them in Volcano Planner can cause infinite loops - CALCITE-1271

  was:
FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule are leveraged in 
DRILL-3855.
They are used in HepPlanner, but if they additionally will be enabled in main 
logical planning stage for Volcano planner, more cases will be covered with 
these rules.
For example: 
{code}
WITH year_total_1
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c),
 year_total_2
 AS (SELECT c.r_regionkeycustomer_id,
1 year_total
 FROM   cp.`tpch/region.parquet` c
 UNION ALL
 SELECT c.n_nationkeycustomer_id,
1 year_total
 FROM   cp.`tpch/nation.parquet` c)
SELECT count(t_w_firstyear.customer_id) as ct
FROM   year_total_1 t_w_firstyear,
   year_total_2 t_w_secyear
WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
 AND t_w_firstyear.year_total > 0 and t_w_secyear.year_total > 0
{code}

Currently using them in Volcano Planner can cause infinite loops - CALCITE-1271


> Use FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule in main logical 
> stage
> --
>
> Key: DRILL-6371
> URL: https://issues.apache.org/jira/browse/DRILL-6371
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects Versions: 1.13.0
>Reporter: Vitalii Diravka
>Priority: Minor
> Fix For: Future
>
>
> FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule are leveraged in 
> DRILL-3855.
> They are used in HepPlanner, but if they additionally will be enabled in main 
> logical planning stage for Volcano planner, more cases will be covered with 
> these rules.
> For example: 
> {code}
> WITH year_total_1
>  AS (SELECT c.r_regionkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/region.parquet` c
>  UNION ALL
>  SELECT c.n_nationkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/nation.parquet` c),
>  year_total_2
>  AS (SELECT c.r_regionkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/region.parquet` c
>  UNION ALL
>  SELECT c.n_nationkeycustomer_id,
> 1 year_total
>  FROM   cp.`tpch/nation.parquet` c)
> SELECT count(t_w_firstyear.customer_id) as ct
> FROM   year_total_1 t_w_firstyear,
>year_total_2 t_w_secyear
> WHERE  t_w_firstyear.year_total = t_w_secyear.year_total
>  AND 

[jira] [Updated] (DRILL-3855) Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-3855:
---
Description: 
Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
reported in DRILL-3257, FilterSetOpTransposeRule, 
DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
Calcite, we have to enable these two rules to lift the performance. 

In addition, will update the plan validation in Unit test in response of the 
newly enabled rules. 

To cover complex cases (with more than two subqueries) these rules should be 
added to main logical stage for Volcano Planner - DRILL-6371.

  was:
Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
reported in DRILL-3257, FilterSetOpTransposeRule, 
DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
Calcite, we have to enable these two rules to lift the performance. 

In addition, will update the plan validation in Unit test in response of the 
newly enabled rules. 

To cover complex cases these rules should be added to main logical stage for 
Volcano Planner - DRILL-6371.


> Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule
> ---
>
> Key: DRILL-3855
> URL: https://issues.apache.org/jira/browse/DRILL-3855
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects Versions: 1.13.0
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Vitalii Diravka
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
> reported in DRILL-3257, FilterSetOpTransposeRule, 
> DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
> Calcite, we have to enable these two rules to lift the performance. 
> In addition, will update the plan validation in Unit test in response of the 
> newly enabled rules. 
> To cover complex cases (with more than two subqueries) these rules should be 
> added to main logical stage for Volcano Planner - DRILL-6371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-3855) Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-3855:
---
Description: 
Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
reported in DRILL-3257, FilterSetOpTransposeRule, 
DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
Calcite, we have to enable these two rules to lift the performance. 

In addition, will update the plan validation in Unit test in response of the 
newly enabled rules. 

To cover complex cases these rules should be added to main logical stage for 
Volcano Planner - DRILL-6371.

  was:
Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
reported in DRILL-3257, FilterSetOpTransposeRule, 
DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
Calcite, we have to enable these two rules to lift the performance. 

In addition, will update the plan validation in Unit test in response of the 
newly enabled rules. 



> Enable FilterSetOpTransposeRule, DrillProjectSetOpTransposeRule
> ---
>
> Key: DRILL-3855
> URL: https://issues.apache.org/jira/browse/DRILL-3855
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects Versions: 1.13.0
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Vitalii Diravka
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Since the infinite planning issues (Calcite Volcano Planner: Calcite-900) 
> reported in DRILL-3257, FilterSetOpTransposeRule, 
> DrillProjectSetOpTransposeRule were disabled. After it can be resolved in 
> Calcite, we have to enable these two rules to lift the performance. 
> In addition, will update the plan validation in Unit test in response of the 
> newly enabled rules. 
> To cover complex cases these rules should be added to main logical stage for 
> Volcano Planner - DRILL-6371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6173) Support transitive closure during filter push down and partition pruning

[Vitalii Diravka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Diravka updated DRILL-6173:
---
Description: 
There is Calcite rule JoinPushTransitivePredicatesRule but it does not work in 
Drill. 
 Applying it in Drill will allow for equi-join queries to push filter condition 
from one table to another:
{code:sql}
select * 
from A, B 
where
A.id = B.id and 
B.id = 100
{code}
In that case it is possible that Scan operator for A table will not scan all 
data. 
 For table A it can lead for applying: 
 1. [Partition pruning for Hive tables and partiotion/directory pruning for 
file system 
tables|https://drill.apache.org/docs/partition-pruning-introduction/] 
 2. [Parquet filter 
pushdown|https://drill.apache.org/docs/parquet-filter-pushdown/]

 

Note: transitive closure doesn't work for some cases, these Calcite issues can 
resolve them:
CALCITE-1048, CALCITE-2274, CALCITE-2275, CALCITE-2241. They are tracked by 
DRILL-6350

  was:
There is Calcite rule JoinPushTransitivePredicatesRule but it does not work in 
Drill. 
 Applying it in Drill will allow for equi-join queries to push filter condition 
from one table to another:
{code:sql}
select * 
from A, B 
where
A.id = B.id and 
B.id = 100
{code}
In that case it is possible that Scan operator for A table will not scan all 
data. 
 For table A it can lead for applying: 
 1. [Partition pruning for Hive tables and partiotion/directory pruning for 
file system 
tables|https://drill.apache.org/docs/partition-pruning-introduction/] 
 2. [Parquet filter 
pushdown|https://drill.apache.org/docs/parquet-filter-pushdown/]

 

Note: transitive closure doesn't work for some cases, these Calcite issues can 
resolve them:
CALCITE-1048, CALCITE-2274, CALCITE-2275, CALCITE-2241.


> Support transitive closure during filter push down and partition pruning
> 
>
> Key: DRILL-6173
> URL: https://issues.apache.org/jira/browse/DRILL-6173
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Query Planning  Optimization
>Affects Versions: 1.12.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> There is Calcite rule JoinPushTransitivePredicatesRule but it does not work 
> in Drill. 
>  Applying it in Drill will allow for equi-join queries to push filter 
> condition from one table to another:
> {code:sql}
> select * 
> from A, B 
> where
> A.id = B.id and 
> B.id = 100
> {code}
> In that case it is possible that Scan operator for A table will not scan all 
> data. 
>  For table A it can lead for applying: 
>  1. [Partition pruning for Hive tables and partiotion/directory pruning for 
> file system 
> tables|https://drill.apache.org/docs/partition-pruning-introduction/] 
>  2. [Parquet filter 
> pushdown|https://drill.apache.org/docs/parquet-filter-pushdown/]
>  
> Note: transitive closure doesn't work for some cases, these Calcite issues 
> can resolve them:
> CALCITE-1048, CALCITE-2274, CALCITE-2275, CALCITE-2241. They are tracked by 
> DRILL-6350



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)