[jira] [Commented] (DRILL-5973) Support injection of time-bound pauses in server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284634#comment-16284634
 ] 

ASF GitHub Bot commented on DRILL-5973:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1055
  
@parthchandra , the code was already rebased. The failing unit tests were 
related to TestCountDownLatchInjection, which had APIs that were unnecessarily 
extended to incorporate pause durations. Reverting it back to the original 
constructor resolved the issues and both, previously failing unit tests passed. 
Waiting for the rest of the regressions to pass. 


> Support injection of time-bound pauses in server
> 
>
> Key: DRILL-5973
> URL: https://issues.apache.org/jira/browse/DRILL-5973
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
> Fix For: 1.13.0
>
>
> While working on DRILL-3640 , when creating a unit test for a server-induced 
> timeout, the injecting a pause leaves the JUnit framework's DrillClient 
> without a handle to the query on the server. This is because we injected the 
> pause to occur before the server could send back a query ID, so the 
> DrillClient has no way to unpause the server.
> The workaround to support this unit test is to allow for injecting pauses 
> with a defined time-bound, after which the server would resume.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284500#comment-16284500
 ] 

Timothy Farkas commented on DRILL-5967:
---

Yup that's the one [~dechanggu]. Thanks

> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Dechang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284483#comment-16284483
 ] 

Dechang Gu commented on DRILL-5967:
---

Is it this one: c8ca958eeb5e3b314aca9bb451e1631cbde18d83

@ilooner
- Possible fix for memory leak in HashPartitionSender
ilooner committed 17 days ago 


Will try it.

> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284422#comment-16284422
 ] 

Timothy Farkas commented on DRILL-5967:
---

[~dechanggu] I have another proposed fix. I have updated my branch here: 
https://github.com/ilooner/drill/tree/DRILL-5967 . Could you do another run?

I believe the issue is that the partition sender was not waiting for all the 
asynchronously sent batches to be freed before closing the memory allocator.

> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5973) Support injection of time-bound pauses in server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284223#comment-16284223
 ] 

ASF GitHub Bot commented on DRILL-5973:
---

Github user kkhatua commented on a diff in the pull request:

https://github.com/apache/drill/pull/1055#discussion_r155877298
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/testing/CountDownLatchInjectionImpl.java
 ---
@@ -41,8 +41,11 @@
   private CountDownLatchInjectionImpl(@JsonProperty("address") final 
String address,
   @JsonProperty("port") final int port,
   @JsonProperty("siteClass") final 
String siteClass,
-  @JsonProperty("desc") final String 
desc) throws InjectionConfigurationException {
-super(address, port, siteClass, desc, 0, 1);
+  @JsonProperty("desc") final String 
desc,
+  @JsonProperty("nSkip") final int 
nSkip,
+  @JsonProperty("nFire") final int 
nFire,
+  @JsonProperty("msPause") final Long 
msPause) throws InjectionConfigurationException {
--- End diff --

Might have hit it accidentally. Don't think I need a null check. Will walk 
through this once and update it.


> Support injection of time-bound pauses in server
> 
>
> Key: DRILL-5973
> URL: https://issues.apache.org/jira/browse/DRILL-5973
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
> Fix For: 1.13.0
>
>
> While working on DRILL-3640 , when creating a unit test for a server-induced 
> timeout, the injecting a pause leaves the JUnit framework's DrillClient 
> without a handle to the query on the server. This is because we injected the 
> pause to occur before the server could send back a query ID, so the 
> DrillClient has no way to unpause the server.
> The workaround to support this unit test is to allow for injecting pauses 
> with a defined time-bound, after which the server would resume.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5973) Support injection of time-bound pauses in server

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284225#comment-16284225
 ] 

ASF GitHub Bot commented on DRILL-5973:
---

Github user kkhatua commented on the issue:

https://github.com/apache/drill/pull/1055
  
Shouldn't be failing. I think I'll rebase and run pre-commit tests to 
confirm. Thanks!


> Support injection of time-bound pauses in server
> 
>
> Key: DRILL-5973
> URL: https://issues.apache.org/jira/browse/DRILL-5973
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
> Fix For: 1.13.0
>
>
> While working on DRILL-3640 , when creating a unit test for a server-induced 
> timeout, the injecting a pause leaves the JUnit framework's DrillClient 
> without a handle to the query on the server. This is because we injected the 
> pause to occur before the server could send back a query ID, so the 
> DrillClient has no way to unpause the server.
> The workaround to support this unit test is to allow for injecting pauses 
> with a defined time-bound, after which the server would resume.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-6007) web ui index page is refreshed at high pace

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284205#comment-16284205
 ] 

ASF GitHub Bot commented on DRILL-6007:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1062#discussion_r155875821
  
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -185,11 +185,10 @@
   
   

[jira] [Commented] (DRILL-5702) Jdbc Driver Class not found

[Holger Kiel (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284190#comment-16284190
 ] 

Holger Kiel commented on DRILL-5702:


+1 LGTM

Thx!

> Jdbc Driver Class not found
> ---
>
> Key: DRILL-5702
> URL: https://issues.apache.org/jira/browse/DRILL-5702
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.11.0
>Reporter: Holger Kiel
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.12.0
>
> Attachments: drill-jdbc-all-1.12.0-SNAPSHOT.jar
>
>
> Cannot connect to drill cluster after upgrade to new Jar 
> drill-jdbc-all-1.11.0.jar. When replacing Jar file with older release 
> drill-jdbc-all-1.10.0.jar, connection works again. Tested with various client 
> applications:
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> ${package.namespace.prefix}org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
>  not found
> at 
> oadd.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:80)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
> at 
> oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
> at 
> oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
> at 
> oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
> at 
> oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
> at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:244)
> at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
> at 
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at 
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
> at 
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at 
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at 
> oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> {code}
> Workaround is using the old driver version.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Dechang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284170#comment-16284170
 ] 

Dechang Gu commented on DRILL-5967:
---

The query did not complete, it was cancelled after this error. detailed log can 
be found here (on selfhosting, it's too big to upload tto the jira):
MAPRTECH/home/dgu/perf_tests/private-drill-perf-test-framework/concurrency/log/231_5.1.0.37549.GA_f599401_TPCH_QueueOFF_24_20171207_002530:$
 ls -ltr

-rw-r--r--  1 root  wheel   127527377 Dec  7 02:26 
drillbit-231_20171207_002530_ucs-node7.log
-rw-r--r--  1 root  wheel   262923105 Dec  7 02:26 
drillbit-231_20171207_002530_ucs-node6.log


> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (DRILL-5967) Memory leak by HashPartitionSender

[Dechang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284121#comment-16284121
 ] 

Dechang Gu edited comment on DRILL-5967 at 12/8/17 7:54 PM:


This is from concurrency test: 16 clients, each executes 16 TPCH queries on 
SF500 parquet files, no complex types. 



was (Author: dechanggu):
This is from concurrency test: 16 clients, each executes 16 TPCH queries on 
SF100 parquet files, no complex types. 


> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5702) Jdbc Driver Class not found

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284132#comment-16284132
 ] 

ASF GitHub Bot commented on DRILL-5702:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/1063
  
+1 LGTM 


> Jdbc Driver Class not found
> ---
>
> Key: DRILL-5702
> URL: https://issues.apache.org/jira/browse/DRILL-5702
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.11.0
>Reporter: Holger Kiel
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.12.0
>
> Attachments: drill-jdbc-all-1.12.0-SNAPSHOT.jar
>
>
> Cannot connect to drill cluster after upgrade to new Jar 
> drill-jdbc-all-1.11.0.jar. When replacing Jar file with older release 
> drill-jdbc-all-1.10.0.jar, connection works again. Tested with various client 
> applications:
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> ${package.namespace.prefix}org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
>  not found
> at 
> oadd.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:80)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
> at 
> oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
> at 
> oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
> at 
> oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
> at 
> oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
> at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:244)
> at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
> at 
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at 
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
> at 
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at 
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at 
> oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> {code}
> Workaround is using the old driver version.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Dechang Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284121#comment-16284121
 ] 

Dechang Gu commented on DRILL-5967:
---

This is from concurrency test: 16 clients, each executes 16 TPCH queries on 
SF100 parquet files, no complex types. 


> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284124#comment-16284124
 ] 

Timothy Farkas commented on DRILL-5967:
---

Also [~dechanggu] does this happen when the query is run to completion? Or does 
this happen when the Query is cancelled while it's running. If it happens when 
a query is cancelled why it is running the HashPartitionSender can have 
unreleased buffers since data is sent downstream asynchronously.

> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5967) Memory leak by HashPartitionSender

[Timothy Farkas (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284049#comment-16284049
 ] 

Timothy Farkas commented on DRILL-5967:
---

Thanks [~dechanggu]. Can you provide the query you used as well as the type of 
data stored in the parquet file? Are there complex types stored in the file? I 
am suspecting that the value vector for a specific data type is not being 
released correctly.

> Memory leak by HashPartitionSender
> --
>
> Key: DRILL-5967
> URL: https://issues.apache.org/jira/browse/DRILL-5967
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> The error found by [~cch...@maprtech.com] and [~dechanggu]
> {code}
> 2017-10-25 15:43:28,658 [260eec84-7de3-03ec-300f-7fdbc111fb7c:frag:2:9] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> Fragment 2:9
> [Error Id: 7eae6c2a-868c-49f8-aad8-b690243ffe9b on mperf113.qa.lab:31010]
> at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
>  ~[drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:301)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267)
>  [drill-java-exec-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.11.0-mapr.jar:1.11.0-mapr]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_121]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_121]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query. 
> Memory leaked: (9216)
> Allocator(op:2:9:0:HashPartitionSender) 100/9216/12831744/100 
> (res/actual/peak/limit)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-3993) Rebase Drill on Calcite master branch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283867#comment-16283867
 ] 

ASF GitHub Bot commented on DRILL-3993:
---

GitHub user KulykRoman opened a pull request:

https://github.com/apache/drill/pull/1066

DRILL-3993: Changes to support Calcite 1.13

Works with Drill-specific Calcite 1.13 from a branch: 
https://github.com/KulykRoman/incubator-calcite/tree/DRILL-3993.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/KulykRoman/drill DRILL-3993

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1066.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1066


commit 71d8b03d60dcbe4d46f1f12aba82354064c25ef6
Author: Roman Kulyk 
Date:   2017-11-30T16:19:12Z

REVERTED:  DRILL-5089

Dynamically load schema of storage plugin only when needed for every query

commit 1c3cea13c798eb831f9e9911ec144bf4824ac5ad
Author: Roman Kulyk 
Date:   2017-08-29T14:10:24Z

DRILL-3993: Changes to support Calcite 1.13

- fixed all compiling errors (main changes were: Maven changes, chenges 
RelNode -> RelRoot, implementing some new methods from updated interfaces, 
chenges some literals, logger changes);
- fixed unexpected column errors, validation errors and assertion errors 
after Calcite update;
- fixed describe table/schema statement according to updated logic;
- added fixes with time-intervals;
- changed precision of BINARY to 65536 (was 1048576) according to updated 
logic (Calcite overrides bigger precision to own maxPrecision);
- ignored some incorrect tests with DRILL-3244;
- changed "Table not found" message to "Object not found within" according 
to new Calcite changes.

commit 7196b366994f6ebcdcb80ca07701e06955e2b856
Author: Volodymyr Vysotskyi 
Date:   2017-11-03T12:18:09Z

DRILL-3993: Fix unit test failures connected with support Calcite 1.13

- Use root schema as default for describe table statement.
Fix TestOpenTSDBPlugin.testDescribe() and 
TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema() 
unit tests.
- Modify expected results for tests:
TestPreparedStatementProvider.invalidQueryValidationError();
TestProjectPushDown.testTPCH1();
TestProjectPushDown.testTPCH3();
TestStorageBasedHiveAuthorization.selectUser1_db_u0_only();
TestStorageBasedHiveAuthorization.selectUser0_db_u1g1_only()
- Fix TestCTAS.whenTableQueryColumnHasStarAndTableFiledListIsSpecified(), 
TestViewSupport.createViewWhenViewQueryColumnHasStarAndViewFiledListIsSpecified(),
 TestInbuiltHiveUDFs.testIf(), testDisableUtf8SupportInQueryString unit tests.
- Fix UnsupportedOperationException and NPE for jdbc tests.
- Fix AssertionError: Conversion to relational algebra failed to preserve 
datatypes

*DrillCompoundIdentifier:
According to the changes, made in [CALCITE-546], star Identifier is 
replaced by empty string during parsing the query. Since Drill uses its own 
DrillCompoundIdentifier, it should also replace star by empty string before 
creating SqlIdentifier instance to avoid further errors connected with star 
column. see SqlIdentifier.isStar() method.

*SqlConverter:
In [CALCITE-1417] added simplification of expressions which should be 
projected every time when a new project rel node is created using RelBuilder. 
It causes assertion errors connected with types nullability. This hook was set 
to false to avoid project expressions simplification. See usage of this hook 
and RelBuilder.project() method.

In Drill the type nullability of the function depends on only the 
nullability of its arguments. In some cases, a function may return null value 
even if it had non-nullable arguments. When Calice simplifies expressions, it 
checks that the type of the result is the same as the type of the expression. 
Otherwise, makeCast() method is called. But when a function returns null 
literal, this cast does nothing, even when the function has a non-nullable 
type. So to avoid this issue, method makeCast() was overridden.

*DrillAvgVarianceConvertlet:
Problem with sum0 and specific changes in old Calcite (it is CALCITE-777). 
(see HistogramShuttle.visitCall method) Changes were made to avoid changes in 
Calcite.

*SqlConverter, DescribeTableHandler, ShowTablesHandler:
New Calcite tries to combine both default and specified workspaces during 
the query validation. In some cases, for example, when describe table statement 
is used, Calcite tries to find INFORMATION_SCHEMA in the schema used as 
default. When it does not find the schema, it tries to find a table with such 

[jira] [Commented] (DRILL-5435) Using Limit causes Memory Leaked Error since 1.10

[Kyrill Alyoshin (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283736#comment-16283736
 ] 

Kyrill Alyoshin commented on DRILL-5435:


Guys, we still get this issue with Drill 1.11.0. So, I don't think DRILL-5420 
has fixed it.

> Using Limit causes Memory Leaked Error since 1.10
> -
>
> Key: DRILL-5435
> URL: https://issues.apache.org/jira/browse/DRILL-5435
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Parquet
>Affects Versions: 1.10.0
>Reporter: F Méthot
>Assignee: Parth Chandra
>
> Here is the details I can provide:
>   We migrated our production system from Drill 1.9 to 1.10 just 5 days ago. 
> (220 nodes cluster)
> Our log show there was some 900+ queries ran without problem in first 4 days. 
>  (similar queries, that never use the `limit` clause)
> Yesterday we started doing simple adhoc select * ... limit 10 queries (like 
> we often do, that was our first use of limit with 1.10)
> and we got a `Memory was leaked` exception below.
> Also, once we get the error, Most of all subsequent user queries fails with 
> Channel Close Exception. We need to restart Drill to bring it back to normal.
> A day later, I used a similar select * limit 10 queries, and the same thing 
> happen, had to restart Drill.
> In the exception, it was refering to a file (1_0_0.parquet)
> I moved that file to smaller test cluster (12 nodes) and got the error on the 
> first attempt. but I am no longer able to reproduce the issue on that file. 
> Between the 12 and 220 nodes cluster, a different Column name and Row Group 
> Start was listed in the error.
> The parquet file was generated by Drill 1.10.
> I tried the same file with a local drill-embedded 1.9 and 1.10 and had no 
> issue.
> Here is the error (manually typed), if you think of anything obvious, let us 
> know.
> AsyncPageReader - User Error Occured: Exception Occurred while reading from 
> disk (can not read class o.a.parquet.format.PageHeader: java.io.IOException: 
> input stream is closed.)
> File:/1_0_0.parquet
> Column: StringColXYZ
> Row Group Start: 115215476
> [Error Id: ]
>   at UserException.java:544)
>   at 
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.handleAndThrowException(AsynvPageReader.java:199)
>   at 
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.access(AsynvPageReader.java:81)
>   at 
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:483)
>   at 
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:392)
>   at 
> o.a.d.exec.store.parquet.columnreaders.AsyncPageReader.AsyncPageReaderTask.call(AsyncPageReader.java:392)
> ...
> Caused by: java.io.IOException: can not read class 
> org.apache.parquet.format.PageHeader: java.io.IOException: Input Stream is 
> closed.
>at o.a.parquet.format.Util.read(Util.java:216)
>at o.a.parquet.format.Util.readPageHeader(Util.java:65)
>at 
> o.a.drill.exec.store.parquet.columnreaders.AsyncPageReader(AsyncPageReaderTask:430)
> Caused by: parquet.org.apache.thrift.transport.TTransportException: Input 
> stream is closed
>at ...read(TIOStreamTransport.java:129)
>at TTransport.readAll(TTransport.java:84)
>at TCompactProtocol.readByte(TCompactProtocol.java:474)
>at TCompactProtocol.readFieldBegin(TCompactProtocol.java:481)
>at InterningProtocol.readFieldBegin(InterningProtocol.java:158)
>at o.a.parquet.format.PageHeader.read(PageHeader.java:828)
>at o.a.parquet.format.Util.read(Util.java:213)
> Fragment 0:0
> [Error id: ...]
> o.a.drill.common.exception.UserException: SYSTEM ERROR: 
> IllegalStateException: Memory was leaked by query. Memory leaked: (524288)
> Allocator(op:0:0:4:ParquetRowGroupScan) 100/524288/39919616/100
>   at o.a.d.common.exceptions.UserException (UserException.java:544)
>   at 
> o.a.d.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
>   at o.a.d.exec.work.fragment.FragmentExecutor.cleanup( 
> FragmentExecutor.java:160)
>   at o.a.d.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
> ...
> Caused by: IllegalStateException: Memory was leaked by query. Memory leaked: 
> (524288)
>   at o.a.d.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
>   at o.a.d.exec.ops.OperatorContextImpl(OperatorContextImpl.java:149)
>   at o.a.d.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
>   at o.a.d.exec.ops.FragmentContext.close(FragmentContext.java:411)
>   at 
> o.a.d.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
>   at 
> o.a.d.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
> This fixed the problem:
> alter 

[jira] [Commented] (DRILL-5919) Add non-numeric support for JSON processing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283686#comment-16283686
 ] 

ASF GitHub Bot commented on DRILL-5919:
---

Github user vladimirtkach commented on the issue:

https://github.com/apache/drill/pull/1026
  
@paul-rogers Made changes, please review. 


> Add non-numeric support for JSON processing
> ---
>
> Key: DRILL-5919
> URL: https://issues.apache.org/jira/browse/DRILL-5919
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>  Labels: doc-impacting
> Fix For: Future
>
>
> Add session options to allow drill working with non standard json strings 
> number literals like: NaN, Infinity, -Infinity. By default these options will 
> be switched off, the user will be able to toggle them during working session.
> *For documentation*
> 1. Added two session options {{store.json.reader.non_numeric_numbers}} and 
> {{store.json.reader.non_numeric_numbers}} that allow to read/write NaN and 
> Infinity as numbers. By default these options are set to false.
> 2. Extended signature of {{convert_toJSON}} and {{convert_fromJSON}} 
> functions by adding second optional parameter that enables read/write NaN and 
> Infinity.
> For example:
> {noformat}
> select convert_fromJSON('{"key": NaN}') from (values(1)); will result with 
> JsonParseException, but
> select convert_fromJSON('{"key": NaN}', true) from (values(1)); will parse 
> NaN as a number.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-6017) Fix for SHUTDOWN button being visible for non Admin users

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283492#comment-16283492
 ] 

ASF GitHub Bot commented on DRILL-6017:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1064#discussion_r155772067
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java 
---
@@ -189,7 +189,7 @@ public ClusterInfo getClusterInfoJSON() {
 String adminUsers = 
ExecConstants.ADMIN_USERS_VALIDATOR.getAdminUsers(optionManager);
 String adminUserGroups = 
ExecConstants.ADMIN_USER_GROUPS_VALIDATOR.getAdminUserGroups(optionManager);
 
-final boolean shouldShowUserInfo = isUserLoggedIn &&
+final boolean shouldShowAdminInfo = isUserLoggedIn &&
--- End diff --

I think it is not fully correct to use this option to decide if we can show 
shutdown button or not.
This option is used only when authentication is enabled. When it's not, 
user by default is admin. So with your fix without authentication, this button 
is also absent.


> Fix for SHUTDOWN button being visible for non Admin users
> -
>
> Key: DRILL-6017
> URL: https://issues.apache.org/jira/browse/DRILL-6017
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Karthikeyan Manivannan
>Priority: Blocker
>  Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> DRILL-4286 introduces shutdown button on index page but when authorization is 
> enabled it should be visible only to admin users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-6019) Only admin should be able to access shutdown resources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283490#comment-16283490
 ] 

ASF GitHub Bot commented on DRILL-6019:
---

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/1065

DRILL-6019: Only admin should be able to access shutdown resources an…

…d information about process and admin users

1. Only admin should be able to access shutdown resources via REST API.
2. Modified ClusterInfo to show information about process and admin users 
only when user is logged in and is admin.
3. Added drillbits comparison based on thier adress and ports to check if 
drillbits are the same (DRILL-6006)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-6019

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1065.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1065


commit 2e1162b73fcecf73170d08e48c0a33b88db77ad9
Author: Arina Ielchiieva 
Date:   2017-12-08T11:19:40Z

DRILL-6019: Only admin should be able to access shutdown resources and 
information about process and admin users

1. Only admin should be able to access shutdown resources via REST API.
2. Modified ClusterInfo to show information about process and admin users 
only when user is logged in and is admin.
3. Added drillbits comparison based on thier adress and ports to check if 
drillbits are the same (DRILL-6006)




> Only admin should be able to access shutdown resources
> --
>
> Key: DRILL-6019
> URL: https://issues.apache.org/jira/browse/DRILL-6019
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Critical
>
> DRILL-4286 introduces graceful shutdown but only admin should be able to 
> access shutdown resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (DRILL-6019) Only admin should be able to access shutdown resources

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva reassigned DRILL-6019:
---

Assignee: Arina Ielchiieva

> Only admin should be able to access shutdown resources
> --
>
> Key: DRILL-6019
> URL: https://issues.apache.org/jira/browse/DRILL-6019
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
>Priority: Critical
>
> DRILL-4286 introduces graceful shutdown but only admin should be able to 
> access shutdown resources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (DRILL-5702) Jdbc Driver Class not found

[Vitalii Diravka (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282729#comment-16282729
 ] 

Vitalii Diravka edited comment on DRILL-5702 at 12/8/17 8:32 AM:
-

[~korlawulki] Could you please verify fixed driver from attachments within your 
environment?


was (Author: vitalii):
[~korlawulki] Could you please verify this driver within your environment?

> Jdbc Driver Class not found
> ---
>
> Key: DRILL-5702
> URL: https://issues.apache.org/jira/browse/DRILL-5702
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.11.0
>Reporter: Holger Kiel
>Assignee: Vitalii Diravka
>Priority: Blocker
> Fix For: 1.12.0
>
> Attachments: drill-jdbc-all-1.12.0-SNAPSHOT.jar
>
>
> Cannot connect to drill cluster after upgrade to new Jar 
> drill-jdbc-all-1.11.0.jar. When replacing Jar file with older release 
> drill-jdbc-all-1.10.0.jar, connection works again. Tested with various client 
> applications:
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> ${package.namespace.prefix}org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback
>  not found
> at 
> oadd.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:80)
> at oadd.org.apache.hadoop.security.Groups.(Groups.java:74)
> at 
> oadd.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
> at 
> oadd.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
> at 
> oadd.org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
> at 
> oadd.org.apache.drill.exec.rpc.security.plain.PlainFactory.createAndLoginUser(PlainFactory.java:63)
> at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.authenticate(UserClient.java:244)
> at 
> oadd.org.apache.drill.exec.rpc.user.UserClient.connect(UserClient.java:171)
> at 
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:432)
> at 
> oadd.org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:379)
> at 
> org.apache.drill.jdbc.impl.DrillConnectionImpl.(DrillConnectionImpl.java:158)
> at 
> org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:72)
> at 
> org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
> at 
> oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
> at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:208)
> {code}
> Workaround is using the old driver version.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)