[jira] [Created] (DRILL-4838) Storage plugin directory location visibility in installation guide
Arnaud Linz created DRILL-4838: -- Summary: Storage plugin directory location visibility in installation guide Key: DRILL-4838 URL: https://issues.apache.org/jira/browse/DRILL-4838 Project: Apache Drill Issue Type: Improvement Components: Documentation Affects Versions: 1.7.0 Environment: Windows 7 64 bits Reporter: Arnaud Linz Priority: Minor For security reasons, my system prevents any user (but the administrator) to write at the C:\ level. When following the installation instruction and starting Drill embedded for windows, it tries to create a storage in /tmp (it does not use the environment variables such as TEMP or TMP) and fails. There is a variable sys.store.provider.local.path that can be set to solve that problem, but it lacks visibility in the installation guide (it is documented further in the configuration section). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4795) Nested aggregate windowed query fails - IllegalStateException
[ https://issues.apache.org/jira/browse/DRILL-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413652#comment-15413652 ] ASF GitHub Bot commented on DRILL-4795: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/563 > Nested aggregate windowed query fails - IllegalStateException > -- > > Key: DRILL-4795 > URL: https://issues.apache.org/jira/browse/DRILL-4795 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.8.0 > Environment: 4 node CentOS cluster >Reporter: Khurram Faraaz >Assignee: Gautam Kumar Parai >Priority: Critical > Attachments: tblWnulls.parquet > > > The below two window function queries fail on MapR Drill 1.8.0 commit ID > 34ca63ba > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c1) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: IllegalStateException: This generator does not support > mappings beyond > Fragment 0:0 > [Error Id: b32ed6b0-6b81-4d5f-bce0-e4ea269c5af1 on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c2) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: IllegalStateException: This generator does not support > mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > From drillbit.log > {noformat} > 2016-07-21 11:19:27,778 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 286f503f-9b20-87e3-d7ec-2d3881f29e4a: select avg(sum(c1)) over (partition by > c2) from `tblWnulls.parquet` > ... > 2016-07-21 11:19:27,979 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > This generator does not support mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: This generator does not support mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] > Caused by: java.lang.IllegalStateException: This generator does not support > mappings beyond > at > org.apache.drill.exec.compile.sig.MappingSet.enterChild(MappingSet.java:102) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:188) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1077) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:815) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:795) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47) > ~[drill-logical-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:359) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationV
[jira] [Commented] (DRILL-4795) Nested aggregate windowed query fails - IllegalStateException
[ https://issues.apache.org/jira/browse/DRILL-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413651#comment-15413651 ] ASF GitHub Bot commented on DRILL-4795: --- Github user amansinha100 commented on the issue: https://github.com/apache/drill/pull/563 +1 > Nested aggregate windowed query fails - IllegalStateException > -- > > Key: DRILL-4795 > URL: https://issues.apache.org/jira/browse/DRILL-4795 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.8.0 > Environment: 4 node CentOS cluster >Reporter: Khurram Faraaz >Assignee: Gautam Kumar Parai >Priority: Critical > Attachments: tblWnulls.parquet > > > The below two window function queries fail on MapR Drill 1.8.0 commit ID > 34ca63ba > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c1) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: IllegalStateException: This generator does not support > mappings beyond > Fragment 0:0 > [Error Id: b32ed6b0-6b81-4d5f-bce0-e4ea269c5af1 on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c2) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: IllegalStateException: This generator does not support > mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > From drillbit.log > {noformat} > 2016-07-21 11:19:27,778 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 286f503f-9b20-87e3-d7ec-2d3881f29e4a: select avg(sum(c1)) over (partition by > c2) from `tblWnulls.parquet` > ... > 2016-07-21 11:19:27,979 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > This generator does not support mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: This generator does not support mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] > Caused by: java.lang.IllegalStateException: This generator does not support > mappings beyond > at > org.apache.drill.exec.compile.sig.MappingSet.enterChild(MappingSet.java:102) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:188) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1077) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:815) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:795) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47) > ~[drill-logical-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:359) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr
[jira] [Commented] (DRILL-2330) Add support for nested aggregate expressions for window aggregates
[ https://issues.apache.org/jira/browse/DRILL-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413801#comment-15413801 ] Dechang Gu commented on DRILL-2330: --- The Queries are added to the Drill Perf Test Framework, and test on 1.8.0 master shows the following results: Env: UCS micro cluster MapR 5.1.0.37549.GA, TPCDS SF100 parquet files, Apache Drill 1.8.0 Master (gitid aaf220f) Query time (ms) tpcds query Run 1 Run 2 Run 3 12 DRILL-4525 Validataion Error 20 DRILL-4525 Validataion Error 47 65,515 27,391 26,247 51 DRILL-4811 IllegalArgument 53 7,722 4,310 3,518 57 22,968 15,956 15,578 63 7,2924,012 3,526 89 5,5923,777 3,585 98DRILL-4525 Validataion Error > Add support for nested aggregate expressions for window aggregates > -- > > Key: DRILL-2330 > URL: https://issues.apache.org/jira/browse/DRILL-2330 > Project: Apache Drill > Issue Type: Improvement > Components: Query Planning & Optimization >Affects Versions: 0.8.0 >Reporter: Abhishek Girish >Assignee: Gautam Kumar Parai > Fix For: 1.8.0 > > Attachments: drillbit.log > > > Aggregate expressions currently cannot be nested. > *The following query fails to validate:* > {code:sql} > select avg(sum(i_item_sk)) from item; > {code} > Error: > Query failed: SqlValidatorException: Aggregate expressions cannot be nested > Log attached. > Reference: TPCDS queries (20, 63, 98, ...) fail to execute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4802) NULLS are not first when NULLS FIRST is used with ORDER BY in window definition
[ https://issues.apache.org/jira/browse/DRILL-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Khurram Faraaz updated DRILL-4802: -- Assignee: (was: Khurram Faraaz) > NULLS are not first when NULLS FIRST is used with ORDER BY in window > definition > --- > > Key: DRILL-4802 > URL: https://issues.apache.org/jira/browse/DRILL-4802 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.8.0 > Environment: 4 node CentOS cluster >Reporter: Khurram Faraaz > > NULLS FIRST is not honored when used with ORDER BY inside window definition. > This in a wrong results issue. > MapR Drill 1.8.0 commit ID : 34ca63ba > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select c2, AVG(SUM(c1)) OVER(partition by c2 > order by c2 nulls first) FROM `tblWnulls.parquet` group by c2; > +---++ > | c2 | EXPR$1 | > +---++ > | a | 11152.0| > | b | 41.0 | > | c | 56.0 | > | d | 4.294967315E9 | > | e | 14.0 | > | null | 106.0 | > +---++ > 6 rows selected (0.227 seconds) > {noformat} > {noformat} > postgres=# select c2, AVG(SUM(c1)) OVER(partition by c2 order by c2 nulls > first) FROM t222 group by c2; > c2 | avg > + > | 106. > a | 11152. > b |41. > c |56. > d |4294967315. > e |14. > (6 rows) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition
[ https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413944#comment-15413944 ] ASF GitHub Bot commented on DRILL-4836: --- GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/564 DRILL-4836: ZK Issue during Drillbit startup, possibly due to race condition ZK Issue during Drillbit startup, possibly due to race condition. A change made in February created a race condition if two Drillbits attempt to create the same storage plugin node at the same time. Revised the code to eliminate the race condition by relying on an exception to detect that the node already exists. You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-4836 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/564.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #564 commit 14d8d0e179419e86d86250eae0b7ae0254162ea7 Author: Paul Rogers Date: 2016-08-09T03:16:59Z DRILL-4836 ZK Issue during Drillbit startup, possibly due to race condition. A change made in February created a race condition if two Drillbits attempt to create the same storage plugin node at the same time. Revised the code to eliminate the race condition by relying on an exception to detect that the node already exists. > ZK Issue during Drillbit startup, possibly due to race condition > > > Key: DRILL-4836 > URL: https://issues.apache.org/jira/browse/DRILL-4836 > Project: Apache Drill > Issue Type: Bug > Components: Server >Reporter: Abhishek Girish >Assignee: Paul Rogers > Fix For: 1.8.0 > > > During a parallel launch of Drillbits on a 4 node cluster, I hit this issue > during startup: > {code} > Exception in thread "main" > org.apache.drill.exec.exception.DrillbitStartupException: Failure during > initial startup of Drillbit. > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261) > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257) > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable > to put > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196) > at > org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94) > ... > at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281) > ... 2 more > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191) > ... 7 more > {code} > And similarly, > {code} > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition
[ https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413971#comment-15413971 ] ASF GitHub Bot commented on DRILL-4836: --- Github user sudheeshkatkam commented on the issue: https://github.com/apache/drill/pull/564 +1 > ZK Issue during Drillbit startup, possibly due to race condition > > > Key: DRILL-4836 > URL: https://issues.apache.org/jira/browse/DRILL-4836 > Project: Apache Drill > Issue Type: Bug > Components: Server >Reporter: Abhishek Girish >Assignee: Paul Rogers > Fix For: 1.8.0 > > > During a parallel launch of Drillbits on a 4 node cluster, I hit this issue > during startup: > {code} > Exception in thread "main" > org.apache.drill.exec.exception.DrillbitStartupException: Failure during > initial startup of Drillbit. > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261) > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257) > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable > to put > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196) > at > org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94) > ... > at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281) > ... 2 more > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191) > ... 7 more > {code} > And similarly, > {code} > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-4795) Nested aggregate windowed query fails - IllegalStateException
[ https://issues.apache.org/jira/browse/DRILL-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautam Kumar Parai resolved DRILL-4795. --- Resolution: Fixed Closed with commit: 0bac42dec63a46ca787f6c5fe5a51b9a97e0d6cc > Nested aggregate windowed query fails - IllegalStateException > -- > > Key: DRILL-4795 > URL: https://issues.apache.org/jira/browse/DRILL-4795 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.8.0 > Environment: 4 node CentOS cluster >Reporter: Khurram Faraaz >Assignee: Gautam Kumar Parai >Priority: Critical > Attachments: tblWnulls.parquet > > > The below two window function queries fail on MapR Drill 1.8.0 commit ID > 34ca63ba > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c1) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: IllegalStateException: This generator does not support > mappings beyond > Fragment 0:0 > [Error Id: b32ed6b0-6b81-4d5f-bce0-e4ea269c5af1 on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select avg(sum(c1)) over (partition by c2) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: IllegalStateException: This generator does not support > mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > From drillbit.log > {noformat} > 2016-07-21 11:19:27,778 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 286f503f-9b20-87e3-d7ec-2d3881f29e4a: select avg(sum(c1)) over (partition by > c2) from `tblWnulls.parquet` > ... > 2016-07-21 11:19:27,979 [286f503f-9b20-87e3-d7ec-2d3881f29e4a:frag:0:0] ERROR > o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: > This generator does not support mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > IllegalStateException: This generator does not support mappings beyond > Fragment 0:0 > [Error Id: ef9056c7-3989-427e-b180-b48741bfc6a4 on centos-01.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:318) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:185) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:287) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) > [drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] > Caused by: java.lang.IllegalStateException: This generator does not support > mappings beyond > at > org.apache.drill.exec.compile.sig.MappingSet.enterChild(MappingSet.java:102) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:188) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1077) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:815) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:795) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47) > ~[drill-logical-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:359) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitUnknown(EvaluationVisitor.java:341
[jira] [Resolved] (DRILL-4796) AssertionError - Nested sum(avg(c1)) over window
[ https://issues.apache.org/jira/browse/DRILL-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gautam Kumar Parai resolved DRILL-4796. --- Resolution: Fixed Closed with commit: 0bac42dec63a46ca787f6c5fe5a51b9a97e0d6cc > AssertionError - Nested sum(avg(c1)) over window > > > Key: DRILL-4796 > URL: https://issues.apache.org/jira/browse/DRILL-4796 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.8.0 >Reporter: Khurram Faraaz >Assignee: Gautam Kumar Parai > > Nested window function query fails on MapR Drill 1.8.0 commit ID 34ca63ba > {noformat} > 0: jdbc:drill:schema=dfs.tmp> select sum(avg(c1)) over (partition by c2) from > `tblWnulls.parquet`; > Error: SYSTEM ERROR: AssertionError: todo: implement syntax > FUNCTION_STAR(COUNT($1)) > [Error Id: fa5e1751-87a2-4880-baf9-7e132253be7c on centos-01.qa.lab:31010] > (state=,code=0) > {noformat} > stack trace from drillbit.log > {noformat} > 2016-07-21 11:25:40,023 [286f4ecc-59bd-113e-1edf-d93411b255aa:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query id > 286f4ecc-59bd-113e-1edf-d93411b255aa: select sum(avg(c1)) over (partition by > c2) from `tblWnulls.parquet` > ... > 2016-07-21 11:25:40,183 [286f4ecc-59bd-113e-1edf-d93411b255aa:foreman] ERROR > o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR: AssertionError: todo: > implement syntax FUNCTION_STAR(COUNT($1)) > [Error Id: fa5e1751-87a2-4880-baf9-7e132253be7c on centos-01.qa.lab:31010] > org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: > AssertionError: todo: implement syntax FUNCTION_STAR(COUNT($1)) > [Error Id: fa5e1751-87a2-4880-baf9-7e132253be7c on centos-01.qa.lab:31010] > at > org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543) > ~[drill-common-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:791) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:901) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:271) > [drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_101] > Caused by: org.apache.drill.exec.work.foreman.ForemanException: > Unexpected exception during fragment initialization: todo: implement syntax > FUNCTION_STAR(COUNT($1)) > ... 4 common frames omitted > Caused by: java.lang.AssertionError: todo: implement syntax > FUNCTION_STAR(COUNT($1)) > at > org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:198) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) > ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14] > at > org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.doFunction(DrillOptiq.java:205) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:105) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.logical.DrillOptiq$RexToDrill.visitCall(DrillOptiq.java:80) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at org.apache.calcite.rex.RexCall.accept(RexCall.java:107) > ~[calcite-core-1.4.0-drill-r14.jar:1.4.0-drill-r14] > at > org.apache.drill.exec.planner.logical.DrillOptiq.toDrill(DrillOptiq.java:77) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.common.DrillProjectRelBase.getProjectExpressions(DrillProjectRelBase.java:111) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.physical.ProjectPrel.getPhysicalOperator(ProjectPrel.java:59) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.physical.SortPrel.getPhysicalOperator(SortPrel.java:81) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT] > at > org.apache.drill.exec.planner.physical.SelectionVectorRemoverPrel.getPhysicalOperator(SelectionVectorRemoverPrel.java:48) > ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.
[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads
[ https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414052#comment-15414052 ] ASF GitHub Bot commented on DRILL-4766: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/561 > FragmentExecutor should use EventProcessor and avoid blocking rpc threads > - > > Key: DRILL-4766 > URL: https://issues.apache.org/jira/browse/DRILL-4766 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Affects Versions: 1.7.0 >Reporter: Deneche A. Hakim >Assignee: Sudheesh Katkam >Priority: Minor > > Currently, rpc thread can block when trying to deliver a cancel or early > termination message to a blocked fragment executor. > Foreman already uses an EventProcessor to avoid such scenarios. > FragmentExecutor could be improved to avoid blocking rpc threads as well -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition
[ https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414051#comment-15414051 ] ASF GitHub Bot commented on DRILL-4836: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/564 > ZK Issue during Drillbit startup, possibly due to race condition > > > Key: DRILL-4836 > URL: https://issues.apache.org/jira/browse/DRILL-4836 > Project: Apache Drill > Issue Type: Bug > Components: Server >Reporter: Abhishek Girish >Assignee: Paul Rogers > Fix For: 1.8.0 > > > During a parallel launch of Drillbits on a 4 node cluster, I hit this issue > during startup: > {code} > Exception in thread "main" > org.apache.drill.exec.exception.DrillbitStartupException: Failure during > initial startup of Drillbit. > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261) > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257) > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable > to put > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196) > at > org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94) > ... > at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281) > ... 2 more > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191) > ... 7 more > {code} > And similarly, > {code} > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4822) Extend distrib-env.sh search to consider site directory
[ https://issues.apache.org/jira/browse/DRILL-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-4822: --- Fix Version/s: 1.8.0 > Extend distrib-env.sh search to consider site directory > --- > > Key: DRILL-4822 > URL: https://issues.apache.org/jira/browse/DRILL-4822 > Project: Apache Drill > Issue Type: Improvement >Reporter: Paul Rogers >Priority: Minor > Fix For: 1.8.0 > > > DRILL-4581 provided revisions to the Drill launch scripts. As part of that > fix, we introduced a new distrib-env.sh file to hold settings created by > custom Drill installers (that is, by custom distributions.) The original > version of this feature looks for distrib-env.sh only in $DRILL_HOME/env. > Experience suggests that installers will write site-specific values to > distrib-env.sh and so the file must then be copied to $DRILL_SITE when > running under YARN. Add $DRILL_SITE to the search path in drill-config.sh for > distrib-env.sh. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4499) Remove unused classes
[ https://issues.apache.org/jira/browse/DRILL-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-4499: --- Fix Version/s: 1.8.0 > Remove unused classes > - > > Key: DRILL-4499 > URL: https://issues.apache.org/jira/browse/DRILL-4499 > Project: Apache Drill > Issue Type: Task >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > Fix For: 1.8.0 > > > List of unused classes that I've tracked over time: > exec/interpreter/src/test/java/org/apache/drill/exec/expr/TestPrune.java > exec/java-exec/src/main/java/org/apache/drill/exec/expr/annotations/MethodMap.java > exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionBody.java > exec/java-exec/src/main/java/org/apache/drill/exec/ops/Multitimer.java > exec/java-exec/src/main/java/org/apache/drill/exec/ops/QuerySetup.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/AvailabilityListener.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControlCommand.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/SendProgress.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/SendProgress.java > exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/DrillUser.java > exec/java-exec/src/main/java/org/apache/drill/exec/store/RecordRecorder.java > exec/java-exec/src/main/java/org/apache/drill/exec/store/schedule/PartialWork.java > exec/java-exec/src/main/java/org/apache/drill/exec/util/AtomicState.java > exec/java-exec/src/main/java/org/apache/drill/exec/work/RecordOutputStream.java > exec/java-exec/src/main/java/org/apache/drill/exec/work/ResourceRequest.java > exec/java-exec/src/main/java/org/apache/drill/exec/work/RootNodeDriver.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4623) Disable Epoll by Default
[ https://issues.apache.org/jira/browse/DRILL-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-4623: --- Fix Version/s: 1.8.0 > Disable Epoll by Default > > > Key: DRILL-4623 > URL: https://issues.apache.org/jira/browse/DRILL-4623 > Project: Apache Drill > Issue Type: Bug >Reporter: Sudheesh Katkam >Assignee: Sudheesh Katkam > Fix For: 1.8.0 > > > At higher concurrency (and/or spuriously), we hit [netty issue > #3539|https://github.com/netty/netty/issues/3539]. This is an issue with the > version of Netty Drill currently uses. Once Drill moves to a later version of > Netty, epoll should be reenabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (DRILL-4581) Various problems in the Drill startup scripts
[ https://issues.apache.org/jira/browse/DRILL-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam reopened DRILL-4581: > Various problems in the Drill startup scripts > - > > Key: DRILL-4581 > URL: https://issues.apache.org/jira/browse/DRILL-4581 > Project: Apache Drill > Issue Type: Sub-task > Components: Server >Affects Versions: 1.6.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.8.0 > > > Noticed the following in drillbit.sh: > 1) Comment: DRILL_LOG_DIRWhere log files are stored. PWD by default. > Code: DRILL_LOG_DIR=/var/log/drill or, if it does not exist, $DRILL_HOME/log > 2) Comment: DRILL_PID_DIRThe pid files are stored. /tmp by default. > Code: DRILL_PID_DIR=$DRILL_HOME > 3) Redundant checking of JAVA_HOME. drillbit.sh sources drill-config.sh which > checks JAVA_HOME. Later, drillbit.sh checks it again. The second check is > both unnecessary and prints a less informative message than the > drill-config.sh check. Suggestion: Remove the JAVA_HOME check in drillbit.sh. > 4) Though drill-config.sh carefully checks JAVA_HOME, it does not export the > JAVA_HOME variable. Perhaps this is why drillbit.sh repeats the check? > Recommended: export JAVA_HOME from drill-config.sh. > 5) Both drillbit.sh and the sourced drill-config.sh check DRILL_LOG_DIR and > set the default value. Drill-config.sh defaults to /var/log/drill, or if that > fails, to $DRILL_HOME/log. Drillbit.sh just sets /var/log/drill and does not > handle the case where that directory is not writable. Suggested: remove the > check in drillbit.sh. > 6) Drill-config.sh checks the writability of the DRILL_LOG_DIR by touching > sqlline.log, but does not delete that file, leaving a bogus, empty client log > file on the drillbit server. Recommendation: use bash commands instead. > 7) The implementation of the above check is a bit awkward. It has a fallback > case with somewhat awkward logic. Clean this up. > 8) drillbit.sh, but not drill-config.sh, attempts to create /var/log/drill if > it does not exist. Recommended: decide on a single choice, implement it in > drill-config.sh. > 9) drill-config.sh checks if $DRILL_CONF_DIR is a directory. If not, defaults > it to $DRILL_HOME/conf. This can lead to subtle errors. If I use > drillbit.sh --config /misspelled/path > where I mistype the path, I won't get an error, I get the default config, > which may not at all be what I want to run. Recommendation: if the value of > DRILL_CONF_DRILL is passed into the script (as a variable or via --config), > then that directory must exist. Else, use the default. > 10) drill-config.sh exports, but may not set, HADOOP_HOME. This may be left > over from the original Hadoop script that the Drill script was based upon. > Recomendation: export only in the case that HADOOP_HOME is set for cygwin. > 11) Drill-config.sh checks JAVA_HOME and prints a big, bold error message to > stderr if JAVA_HOME is not set. Then, it checks the Java version and prints a > different message (to stdout) if the version is wrong. Recommendation: use > the same format (and stderr) for both. > 12) Similarly, other Java checks later in the script produce messages to > stdout, not stderr. > 13) Drill-config.sh searches $JAVA_HOME to find java/java.exe and verifies > that it is executable. The script then throws away what we just found. Then, > drill-bit.sh tries to recreate this information as: > JAVA=$JAVA_HOME/bin/java > This is wrong in two ways: 1) it ignores the actual java location and assumes > it, and 2) it does not handle the java.exe case that drill-config.sh > carefully worked out. > Recommendation: export JAVA from drill-config.sh and remove the above line > from drillbit.sh. > 14) drillbit.sh presumably takes extra arguments like this: > drillbit.sh -Dvar0=value0 --config /my/conf/dir start -Dvar1=value1 > -Dvar2=value2 -Dvar3=value3 > The -D bit allows the user to override config variables at the command line. > But, the scripts don't use the values. > A) drill-config.sh consumes --config /my/conf/dir after consuming the leading > arguments: > while [ $# -gt 1 ]; do > if [ "--config" = "$1" ]; then > shift > confdir=$1 > shift > DRILL_CONF_DIR=$confdir > else > # Presume we are at end of options and break > break > fi > done > B) drill-bit.sh will discard the var1: > startStopStatus=$1 <-- grabs "start" > shift > command=drillbit > shift <-- Consumes -Dvar1=value1 > C) Remaining values passed back into drillbit.sh: > args=$@ > nohup $thiscmd internal_start $command $args > D) Second invocation discards -Dvar2=value2 as described above. > E) Remaining values are passed to runbit: > "$DRILL_HOME"/bin/runbit $command "$@" start > F) Where they aga
[jira] [Resolved] (DRILL-4581) Various problems in the Drill startup scripts
[ https://issues.apache.org/jira/browse/DRILL-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam resolved DRILL-4581. Resolution: Fixed Fix Version/s: 1.8.0 > Various problems in the Drill startup scripts > - > > Key: DRILL-4581 > URL: https://issues.apache.org/jira/browse/DRILL-4581 > Project: Apache Drill > Issue Type: Sub-task > Components: Server >Affects Versions: 1.6.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.8.0 > > > Noticed the following in drillbit.sh: > 1) Comment: DRILL_LOG_DIRWhere log files are stored. PWD by default. > Code: DRILL_LOG_DIR=/var/log/drill or, if it does not exist, $DRILL_HOME/log > 2) Comment: DRILL_PID_DIRThe pid files are stored. /tmp by default. > Code: DRILL_PID_DIR=$DRILL_HOME > 3) Redundant checking of JAVA_HOME. drillbit.sh sources drill-config.sh which > checks JAVA_HOME. Later, drillbit.sh checks it again. The second check is > both unnecessary and prints a less informative message than the > drill-config.sh check. Suggestion: Remove the JAVA_HOME check in drillbit.sh. > 4) Though drill-config.sh carefully checks JAVA_HOME, it does not export the > JAVA_HOME variable. Perhaps this is why drillbit.sh repeats the check? > Recommended: export JAVA_HOME from drill-config.sh. > 5) Both drillbit.sh and the sourced drill-config.sh check DRILL_LOG_DIR and > set the default value. Drill-config.sh defaults to /var/log/drill, or if that > fails, to $DRILL_HOME/log. Drillbit.sh just sets /var/log/drill and does not > handle the case where that directory is not writable. Suggested: remove the > check in drillbit.sh. > 6) Drill-config.sh checks the writability of the DRILL_LOG_DIR by touching > sqlline.log, but does not delete that file, leaving a bogus, empty client log > file on the drillbit server. Recommendation: use bash commands instead. > 7) The implementation of the above check is a bit awkward. It has a fallback > case with somewhat awkward logic. Clean this up. > 8) drillbit.sh, but not drill-config.sh, attempts to create /var/log/drill if > it does not exist. Recommended: decide on a single choice, implement it in > drill-config.sh. > 9) drill-config.sh checks if $DRILL_CONF_DIR is a directory. If not, defaults > it to $DRILL_HOME/conf. This can lead to subtle errors. If I use > drillbit.sh --config /misspelled/path > where I mistype the path, I won't get an error, I get the default config, > which may not at all be what I want to run. Recommendation: if the value of > DRILL_CONF_DRILL is passed into the script (as a variable or via --config), > then that directory must exist. Else, use the default. > 10) drill-config.sh exports, but may not set, HADOOP_HOME. This may be left > over from the original Hadoop script that the Drill script was based upon. > Recomendation: export only in the case that HADOOP_HOME is set for cygwin. > 11) Drill-config.sh checks JAVA_HOME and prints a big, bold error message to > stderr if JAVA_HOME is not set. Then, it checks the Java version and prints a > different message (to stdout) if the version is wrong. Recommendation: use > the same format (and stderr) for both. > 12) Similarly, other Java checks later in the script produce messages to > stdout, not stderr. > 13) Drill-config.sh searches $JAVA_HOME to find java/java.exe and verifies > that it is executable. The script then throws away what we just found. Then, > drill-bit.sh tries to recreate this information as: > JAVA=$JAVA_HOME/bin/java > This is wrong in two ways: 1) it ignores the actual java location and assumes > it, and 2) it does not handle the java.exe case that drill-config.sh > carefully worked out. > Recommendation: export JAVA from drill-config.sh and remove the above line > from drillbit.sh. > 14) drillbit.sh presumably takes extra arguments like this: > drillbit.sh -Dvar0=value0 --config /my/conf/dir start -Dvar1=value1 > -Dvar2=value2 -Dvar3=value3 > The -D bit allows the user to override config variables at the command line. > But, the scripts don't use the values. > A) drill-config.sh consumes --config /my/conf/dir after consuming the leading > arguments: > while [ $# -gt 1 ]; do > if [ "--config" = "$1" ]; then > shift > confdir=$1 > shift > DRILL_CONF_DIR=$confdir > else > # Presume we are at end of options and break > break > fi > done > B) drill-bit.sh will discard the var1: > startStopStatus=$1 <-- grabs "start" > shift > command=drillbit > shift <-- Consumes -Dvar1=value1 > C) Remaining values passed back into drillbit.sh: > args=$@ > nohup $thiscmd internal_start $command $args > D) Second invocation discards -Dvar2=value2 as described above. > E) Remaining values are passed to runbit: > "$DRILL_HOME"/b
[jira] [Resolved] (DRILL-4836) ZK Issue during Drillbit startup, possibly due to race condition
[ https://issues.apache.org/jira/browse/DRILL-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam resolved DRILL-4836. Resolution: Fixed Fixed in [0a4c21c|https://github.com/apache/drill/commit/0a4c21cc15329c063f256f6fbf2c4c69a90d9fa1]. > ZK Issue during Drillbit startup, possibly due to race condition > > > Key: DRILL-4836 > URL: https://issues.apache.org/jira/browse/DRILL-4836 > Project: Apache Drill > Issue Type: Bug > Components: Server >Reporter: Abhishek Girish >Assignee: Paul Rogers > Fix For: 1.8.0 > > > During a parallel launch of Drillbits on a 4 node cluster, I hit this issue > during startup: > {code} > Exception in thread "main" > org.apache.drill.exec.exception.DrillbitStartupException: Failure during > initial startup of Drillbit. > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:284) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:261) > at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:257) > Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: unable > to put > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:196) > at > org.apache.drill.exec.store.sys.store.ZookeeperPersistentStore.putIfAbsent(ZookeeperPersistentStore.java:94) > ... > at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:113) > at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:281) > ... 2 more > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/dfs > at > org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:191) > ... 7 more > {code} > And similarly, > {code} > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for /drill/sys.storage_plugins/kudu > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4839) NumberFormatException with union all when a column on one side contains all nulls
Rahul Challapalli created DRILL-4839: Summary: NumberFormatException with union all when a column on one side contains all nulls Key: DRILL-4839 URL: https://issues.apache.org/jira/browse/DRILL-4839 Project: Apache Drill Issue Type: Bug Components: Execution - Relational Operators Affects Versions: 1.7.0 Reporter: Rahul Challapalli git.commit.id.abbrev=ef6e522 When I do a union all between the below 2 queries, I got a number format exception. {code} select l_orderkey, dir0 from `l_3level/1/one` where l_discount > 0.05 limit 5; +-++ | l_orderkey |dir0| +-++ | 1 | 2015-7-13 | | 1 | 2015-7-13 | | 1 | 2015-7-13 | | 1 | 2015-7-13 | | 3 | 2015-7-13 | +-++ 5 rows selected (0.313 seconds) 0: jdbc:drill:zk=10.10.100.190:5181> select l_orderkey, dir0 from `l_3level/1/one/2015-7-12`; +-+---+ | l_orderkey | dir0 | +-+---+ | 1 | null | | 1 | null | | 1 | null | | 1 | null | | 1 | null | | 1 | null | | 2 | null | | 3 | null | | 3 | null | | 3 | null | +-+---+ 10 rows selected (0.25 seconds) {code} I attached the error log and the data set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4839) NumberFormatException with union all when a column on one side contains all nulls
[ https://issues.apache.org/jira/browse/DRILL-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli updated DRILL-4839: - Attachment: l_3level.tgz error.log > NumberFormatException with union all when a column on one side contains all > nulls > - > > Key: DRILL-4839 > URL: https://issues.apache.org/jira/browse/DRILL-4839 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.7.0 >Reporter: Rahul Challapalli > Attachments: error.log, l_3level.tgz > > > git.commit.id.abbrev=ef6e522 > When I do a union all between the below 2 queries, I got a number format > exception. > {code} > select l_orderkey, dir0 from `l_3level/1/one` where l_discount > 0.05 limit 5; > +-++ > | l_orderkey |dir0| > +-++ > | 1 | 2015-7-13 | > | 1 | 2015-7-13 | > | 1 | 2015-7-13 | > | 1 | 2015-7-13 | > | 3 | 2015-7-13 | > +-++ > 5 rows selected (0.313 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> select l_orderkey, dir0 from > `l_3level/1/one/2015-7-12`; > +-+---+ > | l_orderkey | dir0 | > +-+---+ > | 1 | null | > | 1 | null | > | 1 | null | > | 1 | null | > | 1 | null | > | 1 | null | > | 2 | null | > | 3 | null | > | 3 | null | > | 3 | null | > +-+---+ > 10 rows selected (0.25 seconds) > {code} > I attached the error log and the data set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4839) NumberFormatException with union all when a column on one side contains all nulls
[ https://issues.apache.org/jira/browse/DRILL-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414217#comment-15414217 ] Rahul Challapalli commented on DRILL-4839: -- This looks just like a regular schema change error. Raised this issue so that a developer can confirm there is nothing else going on. > NumberFormatException with union all when a column on one side contains all > nulls > - > > Key: DRILL-4839 > URL: https://issues.apache.org/jira/browse/DRILL-4839 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Relational Operators >Affects Versions: 1.7.0 >Reporter: Rahul Challapalli > Attachments: error.log, l_3level.tgz > > > git.commit.id.abbrev=ef6e522 > When I do a union all between the below 2 queries, I got a number format > exception. > {code} > select l_orderkey, dir0 from `l_3level/1/one` where l_discount > 0.05 limit 5; > +-++ > | l_orderkey |dir0| > +-++ > | 1 | 2015-7-13 | > | 1 | 2015-7-13 | > | 1 | 2015-7-13 | > | 1 | 2015-7-13 | > | 3 | 2015-7-13 | > +-++ > 5 rows selected (0.313 seconds) > 0: jdbc:drill:zk=10.10.100.190:5181> select l_orderkey, dir0 from > `l_3level/1/one/2015-7-12`; > +-+---+ > | l_orderkey | dir0 | > +-+---+ > | 1 | null | > | 1 | null | > | 1 | null | > | 1 | null | > | 1 | null | > | 1 | null | > | 2 | null | > | 3 | null | > | 3 | null | > | 3 | null | > +-+---+ > 10 rows selected (0.25 seconds) > {code} > I attached the error log and the data set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4730) Update JDBC DatabaseMetaData implementation to use new Metadata APIs
[ https://issues.apache.org/jira/browse/DRILL-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4730: -- Fix Version/s: (was: 1.8.0) 1.9.0 > Update JDBC DatabaseMetaData implementation to use new Metadata APIs > > > Key: DRILL-4730 > URL: https://issues.apache.org/jira/browse/DRILL-4730 > Project: Apache Drill > Issue Type: Sub-task > Components: Client - JDBC >Reporter: Venki Korukanti >Assignee: Venki Korukanti > Fix For: 1.9.0 > > > DRILL-4728 is going to add support for new metadata APIs. Replace the > INFORMATION_SCHEMA queries used to get the metadata with the new APIs > provided in Java client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3726) Drill is not properly interpreting CRLF (0d0a). CR gets read as content.
[ https://issues.apache.org/jira/browse/DRILL-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-3726: -- Fix Version/s: (was: 1.8.0) 1.9.0 > Drill is not properly interpreting CRLF (0d0a). CR gets read as content. > > > Key: DRILL-3726 > URL: https://issues.apache.org/jira/browse/DRILL-3726 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Text & CSV >Affects Versions: 1.1.0 > Environment: Linux RHEL 6.6, OSX 10.9 >Reporter: Edmon Begoli >Assignee: Arina Ielchiieva > Fix For: 1.9.0 > > Original Estimate: 120h > Remaining Estimate: 120h > > When we query the last attribute of a text file, we get missing characters. > Looking at the row through Drill, a \r is included at the end of the last > attribute. > Looking in a text editor, it's not embedded into that attribute. > I'm thinking that Drill is not interpreting CRLF (0d0a) as a new line, only > the LF, resulting in the CR becoming part of the last attribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4658) cannot specify tab as a fieldDelimiter in table function
[ https://issues.apache.org/jira/browse/DRILL-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4658: -- Fix Version/s: (was: 1.8.0) 1.9.0 > cannot specify tab as a fieldDelimiter in table function > > > Key: DRILL-4658 > URL: https://issues.apache.org/jira/browse/DRILL-4658 > Project: Apache Drill > Issue Type: Bug > Components: SQL Parser >Affects Versions: 1.6.0 > Environment: Mac OS X, Java 8 >Reporter: Vince Gonzalez >Assignee: Arina Ielchiieva > Fix For: 1.9.0 > > > I can't specify a tab delimiter in the table function because it maybe counts > the characters rather than trying to interpret as a character escape code? > {code} > 0: jdbc:drill:zk=local> select columns[0] as a, cast(columns[1] as bigint) as > b from table(dfs.tmp.`sample_cast.tsv`(type => 'text', fieldDelimiter => > '\t', skipFirstLine => true)); > Error: PARSE ERROR: Expected single character but was String: \t > table sample_cast.tsv > parameter fieldDelimiter > SQL Query null > [Error Id: 3efa82e1-3810-4d4a-b23c-32d6658dffcf on 172.30.1.144:31010] > (state=,code=0) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4560) ZKClusterCoordinator does not call DrillbitStatusListener.drillbitRegistered for new bits
[ https://issues.apache.org/jira/browse/DRILL-4560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4560: -- Fix Version/s: (was: 1.8.0) 1.9.0 > ZKClusterCoordinator does not call DrillbitStatusListener.drillbitRegistered > for new bits > - > > Key: DRILL-4560 > URL: https://issues.apache.org/jira/browse/DRILL-4560 > Project: Apache Drill > Issue Type: Sub-task > Components: Server >Affects Versions: 1.6.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.9.0 > > > ZKClusterCoordinator notifies listeners of type DrillbitStatusListener when > drillbits disappear from ZooKeeper. The YARN Application Master (AM) also > needs to know when bits register themselves with ZK. So, ZKClusterCoordinator > should change to detect new Drill-bits, then call > DrillbitStatusListener.drillbitRegistered with the new Drill-bits. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4714) Add metadata and prepared statement APIs to DrillClient<->Drillbit interface
[ https://issues.apache.org/jira/browse/DRILL-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4714: -- Fix Version/s: (was: 1.8.0) 1.9.0 > Add metadata and prepared statement APIs to DrillClient<->Drillbit interface > > > Key: DRILL-4714 > URL: https://issues.apache.org/jira/browse/DRILL-4714 > Project: Apache Drill > Issue Type: New Feature > Components: Metadata >Reporter: Venki Korukanti >Assignee: Venki Korukanti > Fix For: 1.9.0 > > > Currently ODBC/JDBC drivers spawn a set of queries on INFORMATION_SCHEMA for > metadata. Client has to deal with submitting a query, reading query results > and constructing required objects. Sometimes same work is done twice > (planning work in case of prepare statements) to get the metadata and execute > query. Instead we could simplify the client by providing APIs on the client > interface and let the server construct the required objects and send them to > client directly. These APIs provide common info that can be consumed by the > JDBC/ODBC driver. > [Here|https://docs.google.com/document/d/1ytwGDYGbEFuS3_eUEqRikh7wQABFyHjS0fXJ6wr1eSU/edit?usp=sharing] > is the doc explaining the new APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4752) Remove submit_plan script from Drill distribution
[ https://issues.apache.org/jira/browse/DRILL-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4752: -- Fix Version/s: (was: 1.8.0) 1.9.0 > Remove submit_plan script from Drill distribution > - > > Key: DRILL-4752 > URL: https://issues.apache.org/jira/browse/DRILL-4752 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.6.0 >Reporter: Paul Rogers >Priority: Minor > Fix For: 1.9.0 > > > The Drill distribution includes a submit_plan script in $DRILL_HOME/bin. In > Drill's early days, before Drill has a SQL parser and planner, users > submitted query plans directly to Drill. Since that time, the other Drill > scripts have evolved to support configuration directories, memory options, > and more. The old submit_plan script does not include these enhancements. > It appears that the Drill unit tests do not exercise the script, meaning that > we have no way to verify that it still works. > Rather than invest time to "modernize" submit_plan, we propose to drop it > from the distribution. The script will remain in the Git source tree for the > odd case where someone needs it (and wants to enhance the script to use > drill-config.sh, etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4792) Include session options used for a query as part of the profile
[ https://issues.apache.org/jira/browse/DRILL-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4792: -- Fix Version/s: (was: 1.8.0) 1.9.0 > Include session options used for a query as part of the profile > --- > > Key: DRILL-4792 > URL: https://issues.apache.org/jira/browse/DRILL-4792 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.7.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Minor > Fix For: 1.9.0 > > Attachments: no_session_options.JPG, session_options_block.JPG, > session_options_collapsed.JPG, session_options_json.JPG > > > Include session options used for a query as part of the profile. > This will be very useful for debugging/diagnostics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4840) Sqlline prints log output to stdout on startup
Parth Chandra created DRILL-4840: Summary: Sqlline prints log output to stdout on startup Key: DRILL-4840 URL: https://issues.apache.org/jira/browse/DRILL-4840 Project: Apache Drill Issue Type: Bug Reporter: Parth Chandra With the refactoring of drill scripts, sqlline has now started printing logging messages to stdout when it starts up. This messes up some users' scripts that invoke sqlline. See also, DRILL-2798 which was logged because end users had scripts that broke as a result of printing out additional information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-4840) Sqlline prints log output to stdout on startup
[ https://issues.apache.org/jira/browse/DRILL-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers reassigned DRILL-4840: -- Assignee: Paul Rogers > Sqlline prints log output to stdout on startup > -- > > Key: DRILL-4840 > URL: https://issues.apache.org/jira/browse/DRILL-4840 > Project: Apache Drill > Issue Type: Bug >Reporter: Parth Chandra >Assignee: Paul Rogers > > With the refactoring of drill scripts, sqlline has now started printing > logging messages to stdout when it starts up. This messes up some users' > scripts that invoke sqlline. > See also, DRILL-2798 which was logged because end users had scripts that > broke as a result of printing out additional information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4840) Sqlline prints log output to stdout on startup
[ https://issues.apache.org/jira/browse/DRILL-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414486#comment-15414486 ] Paul Rogers commented on DRILL-4840: Preliminary investigation suggests that the "stock" sqlline picks up the proper config and does not log to console. It appears that some of the script changes may be conflicting with the custom setup that Parth is using. In short, the issue may affect developers, but not users of Drill. > Sqlline prints log output to stdout on startup > -- > > Key: DRILL-4840 > URL: https://issues.apache.org/jira/browse/DRILL-4840 > Project: Apache Drill > Issue Type: Bug >Reporter: Parth Chandra >Assignee: Paul Rogers > > With the refactoring of drill scripts, sqlline has now started printing > logging messages to stdout when it starts up. This messes up some users' > scripts that invoke sqlline. > See also, DRILL-2798 which was logged because end users had scripts that > broke as a result of printing out additional information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)