[jira] [Updated] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6435: - Labels: ready-to-commit (was: ) > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6435: - Reviewer: Paul Rogers (was: Vitalii Diravka) > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6436) Store context and name in AbstractStoragePlugin instead of replicating fields in each StoragePlugin
[ https://issues.apache.org/jira/browse/DRILL-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6436: - Labels: ready-to-commit (was: ) > Store context and name in AbstractStoragePlugin instead of replicating fields > in each StoragePlugin > --- > > Key: DRILL-6436 > URL: https://issues.apache.org/jira/browse/DRILL-6436 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488501#comment-16488501 ] ASF GitHub Bot commented on DRILL-5977: --- akumarb2010 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391601936 >> Did you mean that we do not apply predicate pushdown for such conditions? Yes, IMO the users who is applying these predicates on offsets, should be aware of offset scope per partition. So, in such cases where offset predicates without partitionId can be ignored and let it be full scan as these are in valid queries in kafka perspective. Please let me know your thoughts. >> If we do not pushdown, all partitions will be scanned from startOffset to endOffset. Yes, but actually this kind queries itself are in valid except for topics with one partition. So better to not consider them. >> Are there any drawbacks of applying pushdown on such conditions? These are not drawbacks, but my point is let's not handle this invalid queries. >> we can use this predicate pushdown feature for external checkpointing mechanism. Mostly timestamp based predicates will be useful for the users. Only use case which i can see , for offset based predicates will be for checkpointing and all the check pointing strategies used in various systems like Storm/Spark streaming/Camus etc will be checkpointing on offset per partition. >> Even with pushdown we will return empty results. Pushdown is applied to each of the predicates independently and merged. The implementation is such that it ensures offsetsForTimes is called only for valid partitions (i.e., partitions returned by partitionsFor). Hence we will not run into the situation where offsetsForTimes blocks infinitely. Thanks for clarifying it. Good to know that you are already handling these cases. >> I will add test case for this situation. (I have already added cases for predicates with invalid offsets and timestamps.) Thanks @aravi5 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488498#comment-16488498 ] ASF GitHub Bot commented on DRILL-5977: --- akumarb2010 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391601936 >> Did you mean that we do not apply predicate pushdown for such conditions? Yes, IMO the users who is applying these predicates on offsets, should be aware of offset scope per partition. So, in such cases where offset predicates without partitionId can be ignored and let it be full scan as these are in valid queries in kafka perspective. Please let me know your thoughts. >> If we do not pushdown, all partitions will be scanned from startOffset to endOffset. Yes, but actually this kind queries itself are in valid except for topics with one partition. So better to not consider them. >> Are there any drawbacks of applying pushdown on such conditions? These are not drawbacks, but my point is let's not handle this invalid queries. >> we can use this predicate pushdown feature for external checkpointing mechanism. Mostly timestamp based predicates will be useful for the users. Only use case which i can see , for offset based predicates will be for checkpointing and all the check pointing strategies used in various systems like Storm/Spark streaming/Camus etc will be checkpointing on offset per partition. >> Even with pushdown we will return empty results. Pushdown is applied to each of the predicates independently and merged. The implementation is such that it ensures offsetsForTimes is called only for valid partitions (i.e., partitions returned by partitionsFor). Hence we will not run into the situation where offsetsForTimes blocks infinitely. Thanks for clarifying it. Good to know that you are already handling these cases. >> I will add test case for this situation. (I have already added cases for predicates with invalid offsets and timestamps.) Thanks @aravi5 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5990) Apache Drill /status API returns OK ('Running') even with JRE while queries will not work - make status API reflect the fact that Drill is broken on JRE or stop Drill sta
[ https://issues.apache.org/jira/browse/DRILL-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5990: Labels: ready-to-commit (was: ) > Apache Drill /status API returns OK ('Running') even with JRE while queries > will not work - make status API reflect the fact that Drill is broken on JRE > or stop Drill starting up with JRE > --- > > Key: DRILL-5990 > URL: https://issues.apache.org/jira/browse/DRILL-5990 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.10.0, 1.11.0 > Environment: Docker >Reporter: Hari Sekhon >Assignee: Kunal Khatua >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > If running Apache Drill on versions 1.10 / 1.11 on JRE it appears that > queries no longer run without JDK, but the /status monitoring API endpoint > does not reflect the fact the Apache Drill will not work and still returns > 'Running'. > While 'Running' is technically true the process is up, it's effectively > broken and Apache Drill should either reflect this in /status that is is > broken or refuse to start up on JRE altogether. > See this ticket for more information: > https://github.com/HariSekhon/Dockerfiles/pull/15 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488461#comment-16488461 ] ASF GitHub Bot commented on DRILL-6435: --- paul-rogers commented on issue #1286: DRILL-6435: MappingSet is stateful, so it can't be shared between threads URL: https://github.com/apache/drill/pull/1286#issuecomment-391595679 Thanks much for the explanation: I learned something new! +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6435: - Fix Version/s: 1.14.0 > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6179) Added pcapng-format support
[ https://issues.apache.org/jira/browse/DRILL-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6179: - Reviewer: Paul Rogers > Added pcapng-format support > --- > > Key: DRILL-6179 > URL: https://issues.apache.org/jira/browse/DRILL-6179 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.13.0 >Reporter: Vlad >Assignee: Vlad >Priority: Major > Labels: doc-impacting > Fix For: 1.14.0 > > > The _PCAP Next Generation Dump File Format_ (or pcapng for short) [1] is an > attempt to overcome the limitations of the currently widely used (but > limited) libpcap format. > At a first level, it is desirable to query and filter by source and > destination IP and port, and src/dest mac addreses or by protocol. Beyond > that, however, it would be very useful to be able to group packets by TCP > session and eventually to look at packet contents. > Initial work is available at > https://github.com/mapr-demos/drill/tree/pcapng_dev > [1] https://pcapng.github.io/pcapng/ > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6415: - Reviewer: Timothy Farkas > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code} > {code} > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time > elapsed: 180.028 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > {code} -- T
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488353#comment-16488353 ] ASF GitHub Bot commented on DRILL-5977: --- aravi5 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391573236 Thanks @akumarb2010 for your comments. I have a few questions regarding your comments. >In Kafka, offset scope itself is per partition. I am unable to find any use case, where we can take the range of offsets and apply on all partitions. In most of the scenario's they may not be valid offsets. >IMHO, we should only apply predicate pushdown where we have exact scan specs. 1. Take this query for example ``` SELECT * FROM kafka.LogEventStream WHERE kafkaMsgOffset >= 1000 AND kafkaMsgOffset < 2000) ``` Did you mean that we do not apply predicate pushdown for such conditions? If we do not pushdown, all partitions will be scanned from `startOffset` to `endOffset`. Instead, by supporting pushdown for such queries, we are limiting the scan range wherever possible. Are there any drawbacks of applying pushdown on such conditions? Also, pushdown implementation handles cases where specified offsets are invalid. 2. Can you elaborate more on this? > we can use this predicate pushdown feature for external checkpointing mechanism. 3. >And coming to time stamps, my point is, in case of invalid partitionId, query might block indefinitely with this feature. Where as, without this feature, we will return empty results. Even with pushdown we will return empty results. Pushdown is applied to each of the predicates independently and merged. The implementation is such that it ensures `offsetsForTimes` is called only for valid partitions (i.e., partitions returned by `partitionsFor`). Hence we will not run into the situation where `offsetsForTimes` blocks infinitely. I will add test case for this situation. (I have already added cases for predicates with invalid offsets and timestamps.) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-5977: - Reviewer: B Anil Kumar > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-5735) UI options grouping and filtering & Metrics hints
[ https://issues.apache.org/jira/browse/DRILL-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-5735: - Reviewer: Sorabh Hamirwasia > UI options grouping and filtering & Metrics hints > - > > Key: DRILL-5735 > URL: https://issues.apache.org/jira/browse/DRILL-5735 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: Muhammad Gelbana >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > I'm thinking of some UI improvements that could make all the difference for > users trying to optimize low-performing queries. > h2. Options > h3. Grouping > We can organize the options to be grouped by their scope of effect, this will > help users easily locate the options they may need to tune. > h3. Filtering > Since the options are a lot, we can add a filtering mechanism (i.e. string > search or group\scope filtering) so the user can filter out the options he's > not interested in. To provide more benefit than the grouping idea mentioned > above, filtering may include keywords also and not just the option name, > since the user may not be aware of the name of the option he's looking for. > h2. Metrics > I'm referring here to the metrics page and the query execution plan page that > displays the overview section and major\minor fragments metrics. We can show > hints for each metric such as: > # What does it represent in more details. > # What option\scope-of-options to tune (increase ? decrease ?) to improve the > performance reported by this metric. > # May be even provide a small dialog to quickly allow the modification of the > related option(s) to that metric -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6432) Allow to print the visualized query plan only
[ https://issues.apache.org/jira/browse/DRILL-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6432: - Reviewer: Sorabh Hamirwasia > Allow to print the visualized query plan only > - > > Key: DRILL-6432 > URL: https://issues.apache.org/jira/browse/DRILL-6432 > Project: Apache Drill > Issue Type: New Feature > Components: Web Server >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Minor > Fix For: 1.14.0 > > > Provide a convenient way to printing the Visual Query Plan only, instead of > the entire profile page. > This allows for capability in specifying the zoom level when printing large > complex plans that might span multiple pages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488344#comment-16488344 ] ASF GitHub Bot commented on DRILL-6438: --- priteshm commented on issue #1284: DRILL-6438: Remove excess logging form some tests. URL: https://github.com/apache/drill/pull/1284#issuecomment-391571988 @vvysotskyi could you please review this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { >
[jira] [Updated] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6438: - Reviewer: Volodymyr Vysotskyi > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" > }, > "csv" : { > "type" : "text", > "extensions" : [ "csv" ], > "delimiter" : "," > }, > "tsv" : { > "type" : "text", > "extensions" : [ "tsv" ], > "delimiter" : "\t" > }, > "httpd" : { > "type" : "httpd", > "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\"" > }, > "parquet" : { > "type" : "parquet" >
[jira] [Updated] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6438: - Reviewer: (was: Arina Ielchiieva) > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" > }, > "csv" : { > "type" : "text", > "extensions" : [ "csv" ], > "delimiter" : "," > }, > "tsv" : { > "type" : "text", > "extensions" : [ "tsv" ], > "delimiter" : "\t" > }, > "httpd" : { > "type" : "httpd", > "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\"" > }, > "parquet" : { > "type" : "parquet"
[jira] [Updated] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6437: - Reviewer: Vitalii Diravka (was: Arina Ielchiieva) > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > Fix For: 1.14.0 > > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6438: - Fix Version/s: 1.14.0 > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" > }, > "csv" : { > "type" : "text", > "extensions" : [ "csv" ], > "delimiter" : "," > }, > "tsv" : { > "type" : "text", > "extensions" : [ "tsv" ], > "delimiter" : "\t" > }, > "httpd" : { > "type" : "httpd", > "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\"" > }, > "parquet" : { > "type" : "parquet" > }, >
[jira] [Commented] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488336#comment-16488336 ] ASF GitHub Bot commented on DRILL-6437: --- priteshm commented on issue #1285: DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow. URL: https://github.com/apache/drill/pull/1285#issuecomment-391571029 @vdiravka can you please review this change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > Fix For: 1.14.0 > > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488334#comment-16488334 ] ASF GitHub Bot commented on DRILL-6437: --- priteshm commented on issue #1285: DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow. URL: https://github.com/apache/drill/pull/1285#issuecomment-391571029 @parthchandra or @arina-ielchiieva can you please review this change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > Fix For: 1.14.0 > > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6437: - Fix Version/s: 1.14.0 > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > Fix For: 1.14.0 > > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488292#comment-16488292 ] ASF GitHub Bot commented on DRILL-6435: --- vrozov commented on issue #1286: DRILL-6435: MappingSet is stateful, so it can't be shared between threads URL: https://github.com/apache/drill/pull/1286#issuecomment-391560716 > The specific fix may be OK, but I believe it overlooks a broader point. The `MappingSet` is supposed to be a static association of elements used in code generation. This mapping is meant to be static and thus sharable. > If so, then making the objects non-static makes sense; but I wonder what other things will be broken since the original design appeared to be that `MappingSets` are static. I don't see `MappingSet` being designed to be `static` and thus sharable between threads. The state of the object is modified during code generation (expression compilation) and if there are multiple queries submitted for code generation at the same time, the state machine of the `MappingSet` goes wild as a calls to `enterConstant()` may be followed by a call to `enterConstant()` for a different query on a different thread (I am not even worried about multi-thread safety at this point). > If there is an issue, then somewhere we must have added code that alters these objects per-query. We might want to back up and find that use, then ask if it is really necessary. The code that modifies the state of the `MappingSet` object during code generation was always there. It is not a newly introduced code. > At the very least, perhaps we'd want to add a comment explaining why we need per-query modifications rather than the standard, static definitions. `static` definition for `MappingSet` is not standard. In the majority of places it was already declared as instance variable (not `static`). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6436) Store context and name in AbstractStoragePlugin instead of replicating fields in each StoragePlugin
[ https://issues.apache.org/jira/browse/DRILL-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488278#comment-16488278 ] ASF GitHub Bot commented on DRILL-6436: --- Ben-Zvi closed pull request #1282: DRILL-6436: Storage Plugin to have name and context moved to Abstract… URL: https://github.com/apache/drill/pull/1282 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java b/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java index 62f351c458..18428c6907 100644 --- a/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java +++ b/contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseStoragePlugin.java @@ -38,7 +38,6 @@ public class HBaseStoragePlugin extends AbstractStoragePlugin { private static final HBaseConnectionManager hbaseConnectionManager = HBaseConnectionManager.INSTANCE; - private final DrillbitContext context; private final HBaseStoragePluginConfig storeConfig; private final HBaseSchemaFactory schemaFactory; private final HBaseConnectionKey connectionKey; @@ -47,17 +46,13 @@ public HBaseStoragePlugin(HBaseStoragePluginConfig storeConfig, DrillbitContext context, String name) throws IOException { -this.context = context; +super(context, name); this.schemaFactory = new HBaseSchemaFactory(this, name); this.storeConfig = storeConfig; this.name = name; this.connectionKey = new HBaseConnectionKey(); } - public DrillbitContext getContext() { -return this.context; - } - @Override public boolean supportsRead() { return true; diff --git a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java index 1ac1525c22..449a6f90b3 100644 --- a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java +++ b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveStoragePlugin.java @@ -55,14 +55,12 @@ private final HiveStoragePluginConfig config; private HiveSchemaFactory schemaFactory; - private final DrillbitContext context; - private final String name; private final HiveConf hiveConf; - public HiveStoragePlugin(HiveStoragePluginConfig config, DrillbitContext context, String name) throws ExecutionSetupException { + public HiveStoragePlugin(HiveStoragePluginConfig config, DrillbitContext context, String name) + throws ExecutionSetupException { +super(context, name); this.config = config; -this.context = context; -this.name = name; this.hiveConf = createHiveConf(config.getHiveConfigOverride()); this.schemaFactory = new HiveSchemaFactory(this, name, hiveConf); } @@ -75,14 +73,6 @@ public HiveStoragePluginConfig getConfig() { return config; } - public String getName(){ -return name; - } - - public DrillbitContext getContext() { -return context; - } - @Override public HiveScan getPhysicalScan(String userName, JSONOptions selection, List columns) throws IOException { HiveReadEntry hiveReadEntry = selection.getListWith(new ObjectMapper(), new TypeReference(){}); @@ -150,7 +140,7 @@ public synchronized void registerSchemas(SchemaConfig schemaConfig, SchemaPlus p logger.warn("Schema factory forced close failed, error ignored", t); } try { - schemaFactory = new HiveSchemaFactory(this, name, hiveConf); + schemaFactory = new HiveSchemaFactory(this, getName(), hiveConf); } catch (ExecutionSetupException e) { throw new DrillRuntimeException(e); } diff --git a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java index 7a58f7c13b..c38ea3b97b 100755 --- a/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java +++ b/contrib/storage-jdbc/src/main/java/org/apache/drill/exec/store/jdbc/JdbcStoragePlugin.java @@ -77,17 +77,14 @@ private final JdbcStorageConfig config; - private final DrillbitContext context; private final DataSource source; - private final String name; private final SqlDialect dialect; private final DrillJdbcConvention convention; public JdbcStoragePlugin(JdbcStorageConfig config, DrillbitContext context, String name) { -this.context = context; +super(context, name); this.config = config; -
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488268#comment-16488268 ] ASF GitHub Bot commented on DRILL-6435: --- paul-rogers commented on issue #1286: DRILL-6435: MappingSet is stateful, so it can't be shared between threads URL: https://github.com/apache/drill/pull/1286#issuecomment-391554391 The specific fix may be OK, but I believe it overlooks a broader point. The `MappingSet` is supposed to be a static association of elements used in code generation. This mapping is meant to be static and thus sharable. If there is an issue, then somewhere we must have added code that alters these objects per-query. We might want to back up and find that use, then ask if it is really necessary. If so, then making the objects non-static makes sense; but I wonder what other things will be broken since the original design appeared to be that `MappingSet`s are static. At the very least, perhaps we'd want to add a comment explaining why we need per-query modifications rather than the standard, static definitions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-6435: --- Comment: was deleted (was: The specific fix may be OK, but I believe it overlooks a broader point. The {{MappingSet}} is supposed to be a static association of elements used in code generation. This mapping is meant to be static and thus sharable. If there is an issue, then somewhere we must have added code that alters these objects per-query. We might want to back up and find that use, then ask if it is really necessary. If so, then making the objects non-static makes sense. Perhaps we'd want to add a comment explaining why we need per-query modifications rather than the standard, static definitions.) > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488264#comment-16488264 ] Paul Rogers commented on DRILL-6435: The specific fix may be OK, but I believe it overlooks a broader point. The {{MappingSet}} is supposed to be a static association of elements used in code generation. This mapping is meant to be static and thus sharable. If there is an issue, then somewhere we must have added code that alters these objects per-query. We might want to back up and find that use, then ask if it is really necessary. If so, then making the objects non-static makes sense. Perhaps we'd want to add a comment explaining why we need per-query modifications rather than the standard, static definitions. > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488246#comment-16488246 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190438293 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -237,7 +234,7 @@ public void testRestApi() throws Exception { } Assert.assertTrue(listener.isDone()); - Assert.assertEquals(1,drillbitEndpoints.size()); + Assert.assertEquals(2, drillbitEndpoints.size()); Review comment: K thx. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code}
[jira] [Updated] (DRILL-5990) Apache Drill /status API returns OK ('Running') even with JRE while queries will not work - make status API reflect the fact that Drill is broken on JRE or stop Drill sta
[ https://issues.apache.org/jira/browse/DRILL-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5990: Reviewer: Parth Chandra > Apache Drill /status API returns OK ('Running') even with JRE while queries > will not work - make status API reflect the fact that Drill is broken on JRE > or stop Drill starting up with JRE > --- > > Key: DRILL-5990 > URL: https://issues.apache.org/jira/browse/DRILL-5990 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.10.0, 1.11.0 > Environment: Docker >Reporter: Hari Sekhon >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > If running Apache Drill on versions 1.10 / 1.11 on JRE it appears that > queries no longer run without JDK, but the /status monitoring API endpoint > does not reflect the fact the Apache Drill will not work and still returns > 'Running'. > While 'Running' is technically true the process is up, it's effectively > broken and Apache Drill should either reflect this in /status that is is > broken or refuse to start up on JRE altogether. > See this ticket for more information: > https://github.com/HariSekhon/Dockerfiles/pull/15 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6434) Accessing Azure Data Lake Store (HDFS)
[ https://issues.apache.org/jira/browse/DRILL-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488220#comment-16488220 ] Kunal Khatua commented on DRILL-6434: - [~sebclaude], looks like the _*adl*_ filesystem has HDFS client support since Hadoop 2.8.3 (?). Apache Drill is currently tested for Hadoop 2.7.1. You could try building Drill by changing the {{pom.xml}} here: [https://github.com/apache/drill/blob/master/pom.xml#L68] and trying it out. My guess is that it will pretty much work out of the box. If that helps, we can probably update this Jira's title to reflect the subject as {color:#14892c}'Upgrade Hadoop version to 2.8.3 for Accessing Azure Data Lake FS ( {{adl://}} ) '{color} > Accessing Azure Data Lake Store (HDFS) > -- > > Key: DRILL-6434 > URL: https://issues.apache.org/jira/browse/DRILL-6434 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.13.0 >Reporter: SEBASTIEN CLAUDE >Priority: Major > > It would be great if we can access Azure Datalake store with Drill. > > As Azure ADLS is an HDFS storage it would work easily but the uri is adl:// > and not hdfs://, so Drill doesn't support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6433) a process(find) that never finishes slows down apache drill
[ https://issues.apache.org/jira/browse/DRILL-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488215#comment-16488215 ] Kunal Khatua commented on DRILL-6433: - [~mehran] , what is it that you are trying to do with the {{find}} command that you've issued? It's not clear how or why this is a bug in Drill. > a process(find) that never finishes slows down apache drill > --- > > Key: DRILL-6433 > URL: https://issues.apache.org/jira/browse/DRILL-6433 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.13.0 >Reporter: mehran >Priority: Major > > IN version 13 we have a process as follows > find -L / -name java -type f > this process is added to system every day. and it will never finishes. > I have 100 process in my server. > I did not succeed to set JAVA_HOME in drill-env.sh so i set it in /etc/bashrc > these many processes slows down apache drill. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6182) Doc bug on parameter 'drill.exec.spill.fs'
[ https://issues.apache.org/jira/browse/DRILL-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-6182: Fix Version/s: (was: 1.13.0) 1.14.0 > Doc bug on parameter 'drill.exec.spill.fs' > -- > > Key: DRILL-6182 > URL: https://issues.apache.org/jira/browse/DRILL-6182 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Satoshi Yamada >Assignee: Bridget Bevens >Priority: Trivial > Labels: documentation > Fix For: 1.14.0 > > > Parameter 'drill.exe.spill.fs' should be 'drill.exec.spill.fs' (with "c" > after exe).** > Observed in the documents below. > [https://drill.apache.org/docs/start-up-options/] > [https://drill.apache.org/docs/sort-based-and-hash-based-memory-constrained-operators/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6182) Doc bug on parameter 'drill.exec.spill.fs'
[ https://issues.apache.org/jira/browse/DRILL-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-6182: Fix Version/s: 1.13.0 > Doc bug on parameter 'drill.exec.spill.fs' > -- > > Key: DRILL-6182 > URL: https://issues.apache.org/jira/browse/DRILL-6182 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Satoshi Yamada >Assignee: Bridget Bevens >Priority: Trivial > Labels: documentation > Fix For: 1.14.0 > > > Parameter 'drill.exe.spill.fs' should be 'drill.exec.spill.fs' (with "c" > after exe).** > Observed in the documents below. > [https://drill.apache.org/docs/start-up-options/] > [https://drill.apache.org/docs/sort-based-and-hash-based-memory-constrained-operators/] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-5261) Expose REST endpoint in zookeeper
[ https://issues.apache.org/jira/browse/DRILL-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua closed DRILL-5261. --- Resolution: Duplicate Resolved by DRILL-6289 > Expose REST endpoint in zookeeper > - > > Key: DRILL-5261 > URL: https://issues.apache.org/jira/browse/DRILL-5261 > Project: Apache Drill > Issue Type: New Feature >Reporter: Uwe L. Korn >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > It would be nice to also publish the REST API endpoint of each Drillbit in > the Zookeeper. This would mean that we need an additional entry in > {{DrillbitEndpoint}}. While I would know how to add the attribute to the > ProtoBuf definition and filling the attribute with the correct information, > I'm unsure if there is the need for some migration code to support older > {{DrillbitEndpoint}} implementations that don't have this attribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488183#comment-16488183 ] ASF GitHub Bot commented on DRILL-6415: --- dvjyothsna commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190425609 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -237,7 +234,7 @@ public void testRestApi() throws Exception { } Assert.assertTrue(listener.isDone()); - Assert.assertEquals(1,drillbitEndpoints.size()); + Assert.assertEquals(2, drillbitEndpoints.size()); Review comment: Now we are shutting down only one drillbit instead of two. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lan
[jira] [Commented] (DRILL-6443) Search feature for profiles is available only for running OR completed queries, but not both
[ https://issues.apache.org/jira/browse/DRILL-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488173#comment-16488173 ] ASF GitHub Bot commented on DRILL-6443: --- kkhatua commented on issue #1287: DRILL-6443: Enable Search for both running AND completed queries URL: https://github.com/apache/drill/pull/1287#issuecomment-391527798 @arina-ielchiieva since you had reviewed the original PR #1029 for [DRILL-5867](https://issues.apache.org/jira/browse/DRILL-5867), could you do a review of this small change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Search feature for profiles is available only for running OR completed > queries, but not both > > > Key: DRILL-6443 > URL: https://issues.apache.org/jira/browse/DRILL-6443 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.13.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > When running a query in Drill, the {{/profiles}} page will show the search > (and pagination) capabilities only for the top most visible table (i.e. > _Running Queries_ ). > The _Completed Queries_ table will show the search feature only when there > are no running queries. This is because the backend uses a generalized > freemarker macro to define the seach capabilities for the tables being > rendered. With running queries, both, _running_ and _completed queries_ > tables have the same element ID, resulting in the search capability only > being applied to the first table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6443) Search feature for profiles is available only for running OR completed queries, but not both
[ https://issues.apache.org/jira/browse/DRILL-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488171#comment-16488171 ] ASF GitHub Bot commented on DRILL-6443: --- kkhatua commented on issue #1287: DRILL-6443: Enable Search for both running AND completed queries URL: https://github.com/apache/drill/pull/1287#issuecomment-391527560 This is how the search bars appear with running queries: ![image](https://user-images.githubusercontent.com/4335237/40455627-b7cfabb4-5ea2-11e8-8f5e-8e8dbdd21247.png) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Search feature for profiles is available only for running OR completed > queries, but not both > > > Key: DRILL-6443 > URL: https://issues.apache.org/jira/browse/DRILL-6443 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.13.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > When running a query in Drill, the {{/profiles}} page will show the search > (and pagination) capabilities only for the top most visible table (i.e. > _Running Queries_ ). > The _Completed Queries_ table will show the search feature only when there > are no running queries. This is because the backend uses a generalized > freemarker macro to define the seach capabilities for the tables being > rendered. With running queries, both, _running_ and _completed queries_ > tables have the same element ID, resulting in the search capability only > being applied to the first table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6443) Search feature for profiles is available only for running OR completed queries, but not both
[ https://issues.apache.org/jira/browse/DRILL-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488169#comment-16488169 ] ASF GitHub Bot commented on DRILL-6443: --- kkhatua opened a new pull request #1287: DRILL-6443: Enable Search for both running AND completed queries URL: https://github.com/apache/drill/pull/1287 When running a query in Drill, the `/profiles` page will show the search (and pagination) capabilities only for the top most visible table (i.e. Running Queries ). The _Completed Queries_ table will show the search feature only when there are no _Running Queries_. This is because the backend uses a generalized freemarker macro to define the seach capabilities for the tables being rendered. With running queries, both, _Running_ and _Completed_ queries tables have the same element ID, resulting in the search capability only being applied to the first table. This modifies the Freemarker macro to take an additional argument for distinguishing between _Running_ and _Completed_ _Queries_ tables. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Search feature for profiles is available only for running OR completed > queries, but not both > > > Key: DRILL-6443 > URL: https://issues.apache.org/jira/browse/DRILL-6443 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.13.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > When running a query in Drill, the {{/profiles}} page will show the search > (and pagination) capabilities only for the top most visible table (i.e. > _Running Queries_ ). > The _Completed Queries_ table will show the search feature only when there > are no running queries. This is because the backend uses a generalized > freemarker macro to define the seach capabilities for the tables being > rendered. With running queries, both, _running_ and _completed queries_ > tables have the same element ID, resulting in the search capability only > being applied to the first table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6443) Search feature for profiles is available only for running OR completed queries, but not both
[ https://issues.apache.org/jira/browse/DRILL-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-6443: Reviewer: Arina Ielchiieva > Search feature for profiles is available only for running OR completed > queries, but not both > > > Key: DRILL-6443 > URL: https://issues.apache.org/jira/browse/DRILL-6443 > Project: Apache Drill > Issue Type: Bug > Components: Web Server >Affects Versions: 1.13.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > When running a query in Drill, the {{/profiles}} page will show the search > (and pagination) capabilities only for the top most visible table (i.e. > _Running Queries_ ). > The _Completed Queries_ table will show the search feature only when there > are no running queries. This is because the backend uses a generalized > freemarker macro to define the seach capabilities for the tables being > rendered. With running queries, both, _running_ and _completed queries_ > tables have the same element ID, resulting in the search capability only > being applied to the first table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488152#comment-16488152 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190422687 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -163,7 +163,7 @@ public void run() { } }).start(); - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); Review comment: I don't think we need this sleep here since we are already checking the termination condition in the loop below. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488153#comment-16488153 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190422805 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -115,7 +115,7 @@ public void run() { } }).start(); - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); Review comment: Don't need this sleep see other comments below. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code} > {code} > testRestApiShutdown(org.apache.drill.test
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488148#comment-16488148 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190420698 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -237,7 +234,7 @@ public void testRestApi() throws Exception { } Assert.assertTrue(listener.isDone()); - Assert.assertEquals(1,drillbitEndpoints.size()); + Assert.assertEquals(2, drillbitEndpoints.size()); Review comment: Why did we have to change 1 to 2 here? Wasn't this test already passing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] >
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488154#comment-16488154 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190422417 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -201,24 +201,21 @@ public void testRestApi() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = cluster.clientFixture()) { Drillbit drillbit = cluster.drillbit("db1"); - int port = drillbit.getContext().getConfig().getInt("drill.exec.http.port"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int port = drillbit.getWebServerPort(); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); listener = client.queryBuilder().sql(sql).futureSummary(); Thread.sleep(6); - while( port < 8049) { -URL url = new URL("http://localhost:"+port+"/gracefulShutdown";); -HttpURLConnection conn = (HttpURLConnection) url.openConnection(); -conn.setRequestMethod("POST"); -if (conn.getResponseCode() != 200) { - throw new RuntimeException("Failed : HTTP error code : " - + conn.getResponseCode()); -} -port++; + URL url = new URL("http://localhost:"; + port + "/gracefulShutdown"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.setRequestMethod("POST"); + if (conn.getResponseCode() != 200) { +throw new RuntimeException("Failed : HTTP error code : " ++ conn.getResponseCode()); } - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); Review comment: In fact since we are doing the same thing in the other test could you move this code into a helper method? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488150#comment-16488150 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190421300 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -252,44 +249,40 @@ public void testRestApiShutdown() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = cluster.clientFixture()) { Drillbit drillbit = cluster.drillbit("db1"); - int port = drillbit.getContext().getConfig().getInt("drill.exec.http.port"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int port = drillbit.getWebServerPort(); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); listener = client.queryBuilder().sql(sql).futureSummary(); - Thread.sleep(1); - - while( port < 8048) { -URL url = new URL("http://localhost:"+port+"/shutdown";); -HttpURLConnection conn = (HttpURLConnection) url.openConnection(); -conn.setRequestMethod("POST"); -if (conn.getResponseCode() != 200) { - throw new RuntimeException("Failed : HTTP error code : " - + conn.getResponseCode()); + while (true) { +if (listener.isDone()) { + break; } -port++; - } - - Thread.sleep(grace_period); - - Collection drillbitEndpoints = cluster.drillbit() - .getContext() - .getClusterCoordinator() - .getAvailableEndpoints(); +Thread.sleep(100L); + } + URL url = new URL("http://localhost:"; + port + "/shutdown"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.setRequestMethod("POST"); + if (conn.getResponseCode() != 200) { +throw new RuntimeException("Failed : HTTP error code : " ++ conn.getResponseCode()); + } + Thread.sleep(gracePeriod); long currentTime = System.currentTimeMillis(); long stopTime = currentTime + WAIT_TIMEOUT_MS; - while (currentTime < stopTime) { -if (listener.isDone() && drillbitEndpoints.size() == 2) { + while(currentTime < stopTime) { +Collection drillbitEndpoints = cluster.drillbit() +.getContext() +.getClusterCoordinator() +.getAvailableEndpoints(); +if (drillbitEndpoints.size() == 2) { return; } - -Thread.sleep(100L); Review comment: We should keep this sleep. Otherwise we'll hog all the cpu time while we wait for the drillbit to terminate. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488155#comment-16488155 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190422532 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -252,44 +249,40 @@ public void testRestApiShutdown() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = cluster.clientFixture()) { Drillbit drillbit = cluster.drillbit("db1"); - int port = drillbit.getContext().getConfig().getInt("drill.exec.http.port"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int port = drillbit.getWebServerPort(); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); listener = client.queryBuilder().sql(sql).futureSummary(); - Thread.sleep(1); - - while( port < 8048) { -URL url = new URL("http://localhost:"+port+"/shutdown";); -HttpURLConnection conn = (HttpURLConnection) url.openConnection(); -conn.setRequestMethod("POST"); -if (conn.getResponseCode() != 200) { - throw new RuntimeException("Failed : HTTP error code : " - + conn.getResponseCode()); + while (true) { +if (listener.isDone()) { + break; } -port++; - } - - Thread.sleep(grace_period); - - Collection drillbitEndpoints = cluster.drillbit() - .getContext() - .getClusterCoordinator() - .getAvailableEndpoints(); +Thread.sleep(100L); + } + URL url = new URL("http://localhost:"; + port + "/shutdown"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.setRequestMethod("POST"); + if (conn.getResponseCode() != 200) { +throw new RuntimeException("Failed : HTTP error code : " ++ conn.getResponseCode()); + } + Thread.sleep(gracePeriod); Review comment: Since we are already waiting for a termination condition in the loop below, I don't think we need to have this sleep here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488149#comment-16488149 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190420401 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -201,24 +201,21 @@ public void testRestApi() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = cluster.clientFixture()) { Drillbit drillbit = cluster.drillbit("db1"); - int port = drillbit.getContext().getConfig().getInt("drill.exec.http.port"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int port = drillbit.getWebServerPort(); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); listener = client.queryBuilder().sql(sql).futureSummary(); Thread.sleep(6); Review comment: Why do we need to sleep for 60 seconds here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > or
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488151#comment-16488151 ] ASF GitHub Bot commented on DRILL-6415: --- ilooner commented on a change in pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#discussion_r190421685 ## File path: exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java ## @@ -201,24 +201,21 @@ public void testRestApi() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = cluster.clientFixture()) { Drillbit drillbit = cluster.drillbit("db1"); - int port = drillbit.getContext().getConfig().getInt("drill.exec.http.port"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int port = drillbit.getWebServerPort(); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); listener = client.queryBuilder().sql(sql).futureSummary(); Thread.sleep(6); - while( port < 8049) { -URL url = new URL("http://localhost:"+port+"/gracefulShutdown";); -HttpURLConnection conn = (HttpURLConnection) url.openConnection(); -conn.setRequestMethod("POST"); -if (conn.getResponseCode() != 200) { - throw new RuntimeException("Failed : HTTP error code : " - + conn.getResponseCode()); -} -port++; + URL url = new URL("http://localhost:"; + port + "/gracefulShutdown"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.setRequestMethod("POST"); + if (conn.getResponseCode() != 200) { +throw new RuntimeException("Failed : HTTP error code : " ++ conn.getResponseCode()); } - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); Review comment: I don't think we need this sleep here. We can change the loop below to check for the termination condition. ``` while (currentTime < stopTime) { Collection drillbitEndpoints = cluster.drillbit() .getContext() .getClusterCoordinator() .getOnlineEndPoints(); if (drillbitEndpoints.size() == 2) { break; } Thread.sleep(100L); currentTime = System.currentTimeMillis(); } ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.re
[jira] [Updated] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated DRILL-6435: -- Reviewer: Vitalii Diravka > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488137#comment-16488137 ] ASF GitHub Bot commented on DRILL-6435: --- vrozov opened a new pull request #1286: DRILL-6435: MappingSet is stateful, so it can't be shared between threads URL: https://github.com/apache/drill/pull/1286 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488138#comment-16488138 ] ASF GitHub Bot commented on DRILL-6435: --- vrozov commented on issue #1286: DRILL-6435: MappingSet is stateful, so it can't be shared between threads URL: https://github.com/apache/drill/pull/1286#issuecomment-391521051 @vdiravka Please review This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6443) Search feature for profiles is available only for running OR completed queries, but not both
Kunal Khatua created DRILL-6443: --- Summary: Search feature for profiles is available only for running OR completed queries, but not both Key: DRILL-6443 URL: https://issues.apache.org/jira/browse/DRILL-6443 Project: Apache Drill Issue Type: Bug Components: Web Server Affects Versions: 1.13.0 Reporter: Kunal Khatua Assignee: Kunal Khatua Fix For: 1.14.0 When running a query in Drill, the {{/profiles}} page will show the search (and pagination) capabilities only for the top most visible table (i.e. _Running Queries_ ). The _Completed Queries_ table will show the search feature only when there are no running queries. This is because the backend uses a generalized freemarker macro to define the seach capabilities for the tables being rendered. With running queries, both, _running_ and _completed queries_ tables have the same element ID, resulting in the search capability only being applied to the first table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-6437: -- Reviewer: Arina Ielchiieva > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488053#comment-16488053 ] ASF GitHub Bot commented on DRILL-6437: --- ilooner commented on issue #1285: DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow. URL: https://github.com/apache/drill/pull/1285#issuecomment-391499218 @arina-ielchiieva This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6437) Travis Fails Because Logs Are Flooded.
[ https://issues.apache.org/jira/browse/DRILL-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488052#comment-16488052 ] ASF GitHub Bot commented on DRILL-6437: --- ilooner opened a new pull request #1285: DRILL-6437: Removed excess maven logging when downloading dependencies. This fixed Travis failures due to log overflow. URL: https://github.com/apache/drill/pull/1285 The fix was to change the log level maven when downloading artifacts. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Travis Fails Because Logs Are Flooded. > -- > > Key: DRILL-6437 > URL: https://issues.apache.org/jira/browse/DRILL-6437 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.14.0 >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Critical > > The Travis logs are flooded when downloading mysql. > {code} > Downloading from central: > http://repo.maven.apache.org/maven2/com/jcabi/mysql-dist/5.6.14/mysql-dist-5.6.14-linux-amd64.zip > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > Progress (1): 0.1/325 MB > {code} > And the Travis build fails with > {code} > The log length has exceeded the limit of 4 MB (this usually means that the > test suite is raising the same exception over and over). > The job has been terminated > {code} > This doesn't happen in the core apache travis builds because dependencies are > chached on Travis. However, when running a private Travis build that doesn't > have dependencies cached, we have to redownload mysql and we run into this > problem. > Example Travis build with the issue: > https://travis-ci.org/ilooner/drill/builds/382364378 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
issues@drill.apache.org
[ https://issues.apache.org/jira/browse/DRILL-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6419: - Labels: ready-to-commit (was: ) > E2E Integration test for Lateral&Unnest > --- > > Key: DRILL-6419 > URL: https://issues.apache.org/jira/browse/DRILL-6419 > Project: Apache Drill > Issue Type: Task >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
issues@drill.apache.org
[ https://issues.apache.org/jira/browse/DRILL-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia resolved DRILL-6419. -- Resolution: Fixed Reviewer: Parth Chandra Merged with commit id: 17366d3f44451fdfb0b1b2f5f6e55026346aae9b > E2E Integration test for Lateral&Unnest > --- > > Key: DRILL-6419 > URL: https://issues.apache.org/jira/browse/DRILL-6419 > Project: Apache Drill > Issue Type: Task >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6440) Fix ignored unit tests in unnest
[ https://issues.apache.org/jira/browse/DRILL-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia resolved DRILL-6440. -- Resolution: Fixed Reviewer: Sorabh Hamirwasia Fix Version/s: 1.14.0 Merged with commit id: 9b3be792678afeb5f0c87ba4c4a39afe97f20e98 > Fix ignored unit tests in unnest > > > Key: DRILL-6440 > URL: https://issues.apache.org/jira/browse/DRILL-6440 > Project: Apache Drill > Issue Type: Improvement >Reporter: Parth Chandra >Assignee: Parth Chandra >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6440) Fix ignored unit tests in unnest
[ https://issues.apache.org/jira/browse/DRILL-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488033#comment-16488033 ] ASF GitHub Bot commented on DRILL-6440: --- sohami closed pull request #1283: DRILL-6440: Unnest unit tests and fixes for stats URL: https://github.com/apache/drill/pull/1283 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unnest/UnnestRecordBatch.java b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unnest/UnnestRecordBatch.java index 57a0adeb7c..e985c4defe 100644 --- a/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unnest/UnnestRecordBatch.java +++ b/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unnest/UnnestRecordBatch.java @@ -207,6 +207,9 @@ public IterOutcome innerNext() { } finally { stats.stopSetup(); } + // since we never called next on an upstream operator, incoming stats are + // not updated. update input stats explicitly. + stats.batchReceived(0, incoming.getRecordCount(), true); return IterOutcome.OK_NEW_SCHEMA; } else { assert state != BatchState.FIRST : "First batch should be OK_NEW_SCHEMA"; @@ -223,11 +226,13 @@ public IterOutcome innerNext() { context.getExecutorState().fail(ex); return IterOutcome.STOP; } +stats.batchReceived(0, incoming.getRecordCount(), true); return OK_NEW_SCHEMA; } if (lateral.getRecordIndex() == 0) { unnest.resetGroupIndex(); } + stats.batchReceived(0, incoming.getRecordCount(), false); return doWork(); } @@ -348,8 +353,7 @@ protected IterOutcome doWork() { recordCount = 0; final List transfers = Lists.newArrayList(); -final FieldReference fieldReference = -new FieldReference(popConfig.getColumn()); +final FieldReference fieldReference = new FieldReference(popConfig.getColumn()); final TransferPair transferPair = getUnnestFieldTransferPair(fieldReference); diff --git a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestCorrectness.java b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestCorrectness.java index 137966ba33..c04bff7753 100644 --- a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestCorrectness.java +++ b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestCorrectness.java @@ -615,7 +615,8 @@ public void testUnnestNonArrayColumn() { *] * } * - * @see TestResultSetLoaderMapArray TestResultSetLoaderMapArray for similar schema and data + * @see org.apache.drill.exec.physical.rowSet.impl.TestResultSetLoaderMapArray TestResultSetLoaderMapArray for + * similar schema and data * @return TupleMetadata corresponding to the schema */ private TupleMetadata getRepeatedMapSchema() { diff --git a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestWithLateralCorrectness.java b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestWithLateralCorrectness.java index 9318c516b9..f281964fb4 100644 --- a/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestWithLateralCorrectness.java +++ b/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/unnest/TestUnnestWithLateralCorrectness.java @@ -48,7 +48,6 @@ import org.apache.drill.test.rowSet.schema.SchemaBuilder; import org.junit.AfterClass; import org.junit.BeforeClass; -import org.junit.Ignore; import org.junit.Test; import org.junit.experimental.categories.Category; @@ -147,7 +146,6 @@ public void testUnnestVarWidthColumn() { } - @Ignore("RecordBatchSizer throws Exception in RecordBatchSizer.expandMap") @Test public void testUnnestMapColumn() { @@ -297,15 +295,21 @@ public void testUnnestSchemaChange() { } - @Ignore ("Batch limits need to be sync'd with tthe record batch sizer. Fix once the calulations are stabilized") @Test public void testUnnestLimitBatchSize() { -final int limitedOutputBatchSize = 1024; -final int inputBatchSize = 1024+1; -final int limitedOutputBatchSizeBytes = 1024*(4 + 4 + 4 * inputBatchSize); // num rows * (size of int + size of - // int + size of int * num entries in - // array) +final int limitedOutputBatchSize = 127; +final int inputBatchSize = limitedOutputBat
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488008#comment-16488008 ] ASF GitHub Bot commented on DRILL-5270: --- kkhatua commented on issue #1250: DRILL-5270: Improve loading of profiles listing in the WebUI URL: https://github.com/apache/drill/pull/1250#issuecomment-391489764 @arina-ielchiieva I've made the following changes: 1. Refactored to introduce an Archiver 2. Allow for cache to only apply to WebServer 3. For non-webserver request, like SysTables, support for recursive listing. This is because, while archiving speeds up performance for WebServers, SysTables would need access to archived profiles for analytics. 4. Added tests for the ProfileSet cache This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487826#comment-16487826 ] ASF GitHub Bot commented on DRILL-6438: --- ilooner opened a new pull request #1284: DRILL-6438: Remove excess logging form some tests. URL: https://github.com/apache/drill/pull/1284 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" >
[jira] [Updated] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-6438: -- Reviewer: Arina Ielchiieva > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], > "delimiter" : "|" > }, > "csv" : { > "type" : "text", > "extensions" : [ "csv" ], > "delimiter" : "," > }, > "tsv" : { > "type" : "text", > "extensions" : [ "tsv" ], > "delimiter" : "\t" > }, > "httpd" : { > "type" : "httpd", > "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\"" > }, > "parquet" : { > "type" : "parquet" > }, > "json" : { >
[jira] [Commented] (DRILL-6438) Remove excess logging from tests
[ https://issues.apache.org/jira/browse/DRILL-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487827#comment-16487827 ] ASF GitHub Bot commented on DRILL-6438: --- ilooner commented on issue #1284: DRILL-6438: Remove excess logging form some tests. URL: https://github.com/apache/drill/pull/1284#issuecomment-391454738 @arina-ielchiieva This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove excess logging from tests > > > Key: DRILL-6438 > URL: https://issues.apache.org/jira/browse/DRILL-6438 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > TestLocalExchange and TestLoad have this issue. > See example > {code} > Running > org.apache.drill.exec.physical.impl.TestLocalExchange#testGroupByMultiFields > Plan: { > "head" : { > "version" : 1, > "generator" : { > "type" : "ExplainHandler", > "info" : "" > }, > "type" : "APACHE_DRILL_PHYSICAL", > "options" : [ { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.width.max_per_node", > "num_val" : 2, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_mux_exchange", > "bool_val" : true, > "scope" : "SESSION" > }, { > "kind" : "BOOLEAN", > "accessibleScopes" : "ALL", > "name" : "planner.enable_demux_exchange", > "bool_val" : false, > "scope" : "SESSION" > }, { > "kind" : "LONG", > "accessibleScopes" : "ALL", > "name" : "planner.slice_target", > "num_val" : 1, > "scope" : "SESSION" > } ], > "queue" : 0, > "hasResourcePlan" : false, > "resultMode" : "EXEC" > }, > "graph" : [ { > "pop" : "fs-scan", > "@id" : 196611, > "userName" : "travis", > "files" : [ > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/6.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/9.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/3.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/1.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/2.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/7.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/0.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/5.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/4.json", > > "file:/home/travis/build/apache/drill/exec/java-exec/target/org.apache.drill.exec.physical.impl.TestLocalExchange/root/empTable/8.json" > ], > "storage" : { > "type" : "file", > "enabled" : true, > "connection" : "file:///", > "config" : null, > "workspaces" : { > "root" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "tmp" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/dfsTestTmp/1527026062606-0", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > }, > "default" : { > "location" : > "/home/travis/build/apache/drill/exec/java-exec/./target/org.apache.drill.exec.physical.impl.TestLocalExchange/root", > "writable" : true, > "defaultInputFormat" : null, > "allowAccessOutsideWorkspace" : false > } > }, > "formats" : { > "psv" : { > "type" : "text", > "extensions" : [ "tbl" ], >
[jira] [Commented] (DRILL-6436) Store context and name in AbstractStoragePlugin instead of replicating fields in each StoragePlugin
[ https://issues.apache.org/jira/browse/DRILL-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487820#comment-16487820 ] ASF GitHub Bot commented on DRILL-6436: --- ilooner commented on issue #1282: DRILL-6436: Storage Plugin to have name and context moved to Abstract… URL: https://github.com/apache/drill/pull/1282#issuecomment-391453354 @vrozov @paul-rogers Made context and name private final fields. Also removed redundant getContext() methods. And fixed javadoc errors in AbstractStoragePlugin. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Store context and name in AbstractStoragePlugin instead of replicating fields > in each StoragePlugin > --- > > Key: DRILL-6436 > URL: https://issues.apache.org/jira/browse/DRILL-6436 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.14.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6374) Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test
[ https://issues.apache.org/jira/browse/DRILL-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487794#comment-16487794 ] ASF GitHub Bot commented on DRILL-6374: --- Ben-Zvi closed pull request #1262: DRILL-6374: Transitive Closure leads to TPCH Queries regressions and OOM when run concurrency test URL: https://github.com/apache/drill/pull/1262 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java index b5e09ef767..2a79751c61 100644 --- a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java +++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/PlannerPhase.java @@ -265,8 +265,8 @@ static RuleSet getDrillUserConfigurableLogicalRules(OptimizerRulesContext optimi RuleInstance.UNION_TO_DISTINCT_RULE, // Add support for WHERE style joins. - DrillFilterJoinRules.DRILL_FILTER_ON_JOIN, - DrillFilterJoinRules.DRILL_JOIN, + DrillFilterJoinRules.FILTER_INTO_JOIN, + DrillFilterJoinRules.JOIN_PUSH_CONDITION, RuleInstance.JOIN_PUSH_EXPRESSIONS_RULE, // End support for WHERE style joins. @@ -532,19 +532,18 @@ private static RuleSet getSetOpTransposeRules() { /** * RuleSet for join transitive closure, used only in HepPlanner. - * TODO: {@link RuleInstance#JOIN_PUSH_TRANSITIVE_PREDICATES_RULE} should be copied to #staticRuleSet, - * once CALCITE-1048 is solved. This still should be present in {@link #TRANSITIVE_CLOSURE} stage - * for applying additional filters before {@link #DIRECTORY_PRUNING}. + * TODO: {@link RuleInstance#DRILL_JOIN_PUSH_TRANSITIVE_PREDICATES_RULE} should be moved into {@link #staticRuleSet}, + * (with using {@link DrillRelFactories#LOGICAL_BUILDER}) once CALCITE-1048 is solved. This block can be removed then. * * @return set of planning rules */ static RuleSet getJoinTransitiveClosureRules() { return RuleSets.ofList(ImmutableSet. builder() .add( -DrillFilterJoinRules.DRILL_FILTER_ON_JOIN, -DrillFilterJoinRules.DRILL_JOIN, -RuleInstance.JOIN_PUSH_TRANSITIVE_PREDICATES_RULE, -RuleInstance.FILTER_MERGE_RULE +RuleInstance.DRILL_JOIN_PUSH_TRANSITIVE_PREDICATES_RULE, +DrillFilterJoinRules.DRILL_FILTER_INTO_JOIN, +RuleInstance.REMOVE_IS_NOT_DISTINCT_FROM_RULE, +RuleInstance.DRILL_FILTER_MERGE_RULE ).build()); } } diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java index 986fdbd4d1..80bbe88522 100644 --- a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java +++ b/exec/java-exec/src/main/java/org/apache/drill/exec/planner/RuleInstance.java @@ -29,6 +29,7 @@ import org.apache.calcite.rel.rules.AggregateRemoveRule; import org.apache.calcite.rel.rules.FilterCorrelateRule; import org.apache.calcite.rel.rules.FilterMergeRule; +import org.apache.calcite.rel.rules.FilterRemoveIsNotDistinctFromRule; import org.apache.calcite.rel.rules.FilterSetOpTransposeRule; import org.apache.calcite.rel.rules.JoinPushExpressionsRule; import org.apache.calcite.rel.rules.JoinPushThroughJoinRule; @@ -63,6 +64,9 @@ FilterMergeRule FILTER_MERGE_RULE = new FilterMergeRule(DrillRelFactories.LOGICAL_BUILDER); + FilterMergeRule DRILL_FILTER_MERGE_RULE = + new FilterMergeRule(DrillRelBuilder.proto(DrillRelFactories.DRILL_LOGICAL_FILTER_FACTORY)); + FilterCorrelateRule FILTER_CORRELATE_RULE = new FilterCorrelateRule(DrillRelFactories.LOGICAL_BUILDER); @@ -120,6 +124,10 @@ * {@link org.apache.calcite.rel.core.Filter}s if those predicates can be pushed * to its inputs. */ - JoinPushTransitivePredicatesRule JOIN_PUSH_TRANSITIVE_PREDICATES_RULE = - new JoinPushTransitivePredicatesRule(Join.class, DrillRelFactories.LOGICAL_BUILDER); + JoinPushTransitivePredicatesRule DRILL_JOIN_PUSH_TRANSITIVE_PREDICATES_RULE = + new JoinPushTransitivePredicatesRule(Join.class, DrillRelBuilder.proto( + DrillRelFactories.DRILL_LOGICAL_JOIN_FACTORY, DrillRelFactories.DRILL_LOGICAL_FILTER_FACTORY)); + + FilterRemoveIsNotDistinctFromRule REMOVE_IS_NOT_DISTINCT_FROM_RULE = + new FilterRemoveIsNotDistinctFromRule(DrillRelBuilder.proto(DrillRelFactories.DRILL_LOGICAL_FILTER_FACTORY)); } diff --git a/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterJoinRules.java b/exec/java-exec/src/main/java/org
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487786#comment-16487786 ] ASF GitHub Bot commented on DRILL-6415: --- dvjyothsna opened a new pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281 @ilooner Please review This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code} > {code} > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time > elapsed: 180.028 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQ
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487785#comment-16487785 ] ASF GitHub Bot commented on DRILL-6415: --- dvjyothsna commented on issue #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#issuecomment-391446899 Not Able to reopen PR #1263. So Reopening this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code} > {code} > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time > elapsed: 180.028 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > a
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487781#comment-16487781 ] ASF GitHub Bot commented on DRILL-6415: --- dvjyothsna commented on issue #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#issuecomment-391446528 Closing this and reopening PR #1263 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code} > {code} > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time > elapsed: 180.028 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.ut
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487782#comment-16487782 ] ASF GitHub Bot commented on DRILL-6415: --- dvjyothsna closed pull request #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java b/exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java index ccd65e41fe..52f9292428 100644 --- a/exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java +++ b/exec/java-exec/src/test/java/org/apache/drill/test/TestGracefulShutdown.java @@ -103,7 +103,7 @@ public void testOnlineEndPoints() throws Exception { Drillbit drillbit = cluster.drillbit("db2"); DrillbitEndpoint drillbitEndpoint = drillbit.getRegistrationHandle().getEndPoint(); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); new Thread(new Runnable() { public void run() { @@ -115,7 +115,7 @@ public void run() { } }).start(); - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); long currentTime = System.currentTimeMillis(); long stopTime = currentTime + WAIT_TIMEOUT_MS; @@ -151,7 +151,7 @@ public void testStateChange() throws Exception { try (ClusterFixture cluster = builder.build()) { Drillbit drillbit = cluster.drillbit("db2"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); DrillbitEndpoint drillbitEndpoint = drillbit.getRegistrationHandle().getEndPoint(); new Thread(new Runnable() { public void run() { @@ -163,7 +163,7 @@ public void run() { } }).start(); - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); long currentTime = System.currentTimeMillis(); long stopTime = currentTime + WAIT_TIMEOUT_MS; @@ -201,24 +201,21 @@ public void testRestApi() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = cluster.clientFixture()) { Drillbit drillbit = cluster.drillbit("db1"); - int port = drillbit.getContext().getConfig().getInt("drill.exec.http.port"); - int grace_period = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); + int port = drillbit.getWebServerPort(); + int gracePeriod = drillbit.getContext().getConfig().getInt(ExecConstants.GRACE_PERIOD); listener = client.queryBuilder().sql(sql).futureSummary(); Thread.sleep(6); - while( port < 8049) { -URL url = new URL("http://localhost:"+port+"/gracefulShutdown";); -HttpURLConnection conn = (HttpURLConnection) url.openConnection(); -conn.setRequestMethod("POST"); -if (conn.getResponseCode() != 200) { - throw new RuntimeException("Failed : HTTP error code : " - + conn.getResponseCode()); -} -port++; + URL url = new URL("http://localhost:"; + port + "/gracefulShutdown"); + HttpURLConnection conn = (HttpURLConnection) url.openConnection(); + conn.setRequestMethod("POST"); + if (conn.getResponseCode() != 200) { +throw new RuntimeException("Failed : HTTP error code : " ++ conn.getResponseCode()); } - Thread.sleep(grace_period); + Thread.sleep(gracePeriod); Collection drillbitEndpoints = cluster.drillbit() .getContext() .getClusterCoordinator() @@ -237,7 +234,7 @@ public void testRestApi() throws Exception { } Assert.assertTrue(listener.isDone()); - Assert.assertEquals(1,drillbitEndpoints.size()); + Assert.assertEquals(2, drillbitEndpoints.size()); } } @@ -252,44 +249,40 @@ public void testRestApiShutdown() throws Exception { builder = enableWebServer(builder); QueryBuilder.QuerySummaryFuture listener; final String sql = "select * from dfs.root.`.`"; -try ( ClusterFixture cluster = builder.build(); +try (ClusterFixture cluster = builder.build(); final ClientFixture client = clust
[jira] [Updated] (DRILL-6440) Fix ignored unit tests in unnest
[ https://issues.apache.org/jira/browse/DRILL-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sorabh Hamirwasia updated DRILL-6440: - Labels: ready-to-commit (was: ) > Fix ignored unit tests in unnest > > > Key: DRILL-6440 > URL: https://issues.apache.org/jira/browse/DRILL-6440 > Project: Apache Drill > Issue Type: Improvement >Reporter: Parth Chandra >Assignee: Parth Chandra >Priority: Major > Labels: ready-to-commit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6440) Fix ignored unit tests in unnest
[ https://issues.apache.org/jira/browse/DRILL-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487774#comment-16487774 ] ASF GitHub Bot commented on DRILL-6440: --- sohami commented on issue #1283: DRILL-6440: Unnest unit tests and fixes for stats URL: https://github.com/apache/drill/pull/1283#issuecomment-391444730 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix ignored unit tests in unnest > > > Key: DRILL-6440 > URL: https://issues.apache.org/jira/browse/DRILL-6440 > Project: Apache Drill > Issue Type: Improvement >Reporter: Parth Chandra >Assignee: Parth Chandra >Priority: Major > Labels: ready-to-commit > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[ https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487751#comment-16487751 ] ASF GitHub Bot commented on DRILL-6415: --- vrozov commented on issue #1281: DRILL-6415: Fixed TestGracefulShutdown.TestRestApi test from timing out URL: https://github.com/apache/drill/pull/1281#issuecomment-391440323 Please reopen PR #1263 and close this one. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Unit test TestGracefulShutdown.testRestApiShutdown times out > > > Key: DRILL-6415 > URL: https://issues.apache.org/jira/browse/DRILL-6415 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Abhishek Girish >Assignee: Venkata Jyothsna Donapati >Priority: Major > Fix For: 1.14.0 > > > {code} > 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: > -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)): > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > ~[na:1.8.0_161] > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) > ~[na:1.8.0_161] > at > org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203) > ~[classes/:na] > at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242) > ~[classes/:na] > at > org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454) > ~[test-classes/:1.14.0-SNAPSHOT] > at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405) > ~[test-classes/:1.14.0-SNAPSHOT] > at > org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294) > ~[test-classes/:1.14.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.8.0_161] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[na:1.8.0_161] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.8.0_161] > at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161] > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > ~[junit-4.12.jar:4.12] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > ~[junit-4.12.jar:4.12] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70) > ~[jmockit-1.39.jar:1.39] > at > mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34) > ~[jmockit-1.39.jar:1.39] > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > ~[junit-4.12.jar:4.12] > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > ~[junit-4.12.jar:4.12] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[na:1.8.0_161] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161] > {code} > {code} > testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time > elapsed: 180.028 sec <<< ERROR! > org.junit.runners.model.TestTimedOutException: test timed out after 18 > milliseconds > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487680#comment-16487680 ] ASF GitHub Bot commented on DRILL-5977: --- akumarb2010 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391427723 @aravi5 Thanks for providing all the details. Please find my comments as below. >> If the predicates are only on kafkaMsgOffset (for example SELECT * FROM kafka.LogEventStream WHERE kafkaMsgOffset >= 1000 AND kafkaMsgOffset < 2000), this will apply the pushdown to ALL partitions within a topic. If there is a partition where such offsets do not exist (either the offsets have expired or messages for those offsets are yet to be produced), then such partition will not be scanned. In Kafka, `offset` scope itself is per partition. I am unable to find any use case, where we can take the range of offsets and apply on all partitions. In most of the scenario's they may not be valid offsets. IMHO, we should only apply predicate pushdown where we have exact scan specs. For example, in case below scenario, we can apply the predicates without any issues. ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 1 AND kafkaMsgOffset > 1000 AND kafkaMsgOffset < 2000) OR (kafkaPartitionId = 2 AND kafkaMsgOffset > 4000) ``` And this way, we can use this predicate pushdown feature for external checkpointing mechanism. And coming to time stamps, my point is, in case of invalid *partitionId*, query might block indefinitely with this feature. Where as, without this feature, we will return empty results. It will be great if you can add few test cases with invalid partitions like below (Assuming partition 100 doesn't exist) ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 100 AND kafkaMsgTimestamp > 1527092007199) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487673#comment-16487673 ] ASF GitHub Bot commented on DRILL-5977: --- akumarb2010 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391427723 @aravi5 Thanks for providing all the details. Please find my comments as below. >> If the predicates are only on kafkaMsgOffset (for example SELECT * FROM kafka.LogEventStream WHERE kafkaMsgOffset >= 1000 AND kafkaMsgOffset < 2000), this will apply the pushdown to ALL partitions within a topic. If there is a partition where such offsets do not exist (either the offsets have expired or messages for those offsets are yet to be produced), then such partition will not be scanned. In Kafka, `offset` scope itself is per partition. I am unable to find any use case, where we can take the range of offsets and apply on all partitions. In most of the scenario's they may not be valid offsets. IMHO, we should only apply predicate pushdown where we have exact scan specs. For example, in case below scenario, we can apply the predicates without any issues. ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 1 AND kafkaMsgOffset > 1000 AND kafkaMsgOffset < 2000) OR (kafkaPartitionId = 2 AND kafkaMsgOffset > 4000) ``` And this way, we can use this predicate pushdown feature for external checkpointing mechanism. And coming to time stamps, my point in case of invalid partitionId, query might block indefinitely with this feature. Where as in other case, we will return empty results. It will be great if you can add few test cases with invalid partitions like below (Assuming partition 100 doesn't exist) ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 100 AND kafkaMsgTimestamp > 1527092007199) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487671#comment-16487671 ] ASF GitHub Bot commented on DRILL-5977: --- akumarb2010 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391427723 @aravi5 Thanks for providing all the details. Please find my comments as below. >> If the predicates are only on kafkaMsgOffset (for example SELECT * FROM kafka.LogEventStream WHERE kafkaMsgOffset >= 1000 AND kafkaMsgOffset < 2000), this will apply the pushdown to ALL partitions within a topic. If there is a partition where such offsets do not exist (either the offsets have expired or messages for those offsets are yet to be produced), then such partition will not be scanned. In Kafka, `offset` scope itself is per partition. I am unable to find any use case, we can take the range of offsets and apply on all partitions. In most of the scenario's they may not be valid offsets. IMHO, we should only apply predicate pushdown where have exact scan specs. For example, in case below cases we can apply the predicates without any issues. ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 1 AND kafkaMsgOffset > 1000 AND kafkaMsgOffset < 2000) OR (kafkaPartitionId = 2 AND kafkaMsgOffset > 4000) ``` And this way, we can use this predicates feature for external checkpointing mechanism. And coming to time stamps, my point in case of invalid partitionId, query might block indefinitely with this feature. Where as in other case, we will return empty results. It will be great if you can add few test cases with invalid partitions like below (Assuming partition 100 doesn't exist) ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 100 AND kafkaMsgTimestamp > 1527092007199) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487592#comment-16487592 ] ASF GitHub Bot commented on DRILL-5977: --- aravi5 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391413129 @akumarb2010 - user has the flexibility to provide conditions w.r.t any of the metadata fields (`kafkaMsgOffset`, `kafkaMsgTimestamp`, `kafkaPartitionId`). The conditions on these metadata fields is pushed down and translated to a **Scan Spec** which is a collection of partition specific scan spec. Let us consider the following Scan spec before pushdown. ``` KafkaPartitionScanSpec(topicName=TopicTable, partitionId=0, startOffset=0, endOffset=1500) KafkaPartitionScanSpec(topicName=TopicTable, partitionId=1, startOffset=2000, endOffset=3000) KafkaPartitionScanSpec(topicName=TopicTable, partitionId=2, startOffset=25, endOffset=3500) ``` If the predicates are only on `kafkaMsgOffset` (for example `SELECT * FROM kafka.LogEventStream WHERE kafkaMsgOffset >= 1000 AND kafkaMsgOffset < 2000)`, this will apply the pushdown to ALL partitions within a topic. If there is a partition where such offsets do not exist (either the offsets have expired or messages for those offsets are yet to be produced), then such partition will not be scanned. The scan spec in this case would be ``` KafkaPartitionScanSpec(topicName=TopicTable, partitionId=0, startOffset=1000, endOffset=1500) KafkaPartitionScanSpec(topicName=TopicTable, partitionId=1, startOffset=1200, endOffset=2000) KafkaPartitionScanSpec(topicName=TopicTable, partitionId=2, startOffset=1000, endOffset=2000) ``` I am not sure if we should introduce per partition semantics in the query and would prefer keeping it generic. The *scenario you mentioned in (1)* can still be addressed with following predicates ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 1 AND kafkaMsgOffset >= 1000 AND kafkaMsgOffset < 2000) OR (kafkaPartitionId = 2 AND kafkaMsgOffset >= 1500 AND kafkaMsgOffset < 5000) ``` The scan spec in this case would be as follows (notice that there is no `partitionId 0`). ``` KafkaPartitionScanSpec(topicName=TopicTable, partitionId=1, startOffset=1200, endOffset=2000) KafkaPartitionScanSpec(topicName=TopicTable, partitionId=2, startOffset=1000, endOffset=2000) ``` This applies to conditions on `kafkaMsgTimestamp` as well. Most common use case is - user wants to view messages that belong to a specific time window (not very specific about the partition). This can be done by having predicates on `kafkaMsgTimestamp` alone without having to specify `kafkaPartitionId`. ``` SELECT * FROM kafka.LogEventStream WHERE kafkaMsgTimestamp > 1527092007199 AND kafkaMsgOffset < 1527092031717 ``` However user can also specify conditions specific to partition - *scenario mentioned in (2)* ``` SELECT * FROM kafka.LogEventStream WHERE (kafkaPartitionId = 1 AND kafkaMsgTimestamp > 1527092007199 AND kafkaMsgOffset < 1527092031717) OR (kafkaPartitionId = 2 AND kafkaMsgTimestamp > 1527092133501) ``` This would also avoid the problem of calling `offsetsForTimes` on non-existing partitions. Since such partitions are filtered out from the scan spec. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6442) Adjust Hbase disk cost & row count estimation when filter push down is applied
Arina Ielchiieva created DRILL-6442: --- Summary: Adjust Hbase disk cost & row count estimation when filter push down is applied Key: DRILL-6442 URL: https://issues.apache.org/jira/browse/DRILL-6442 Project: Apache Drill Issue Type: Bug Affects Versions: 1.13.0 Reporter: Arina Ielchiieva Assignee: Arina Ielchiieva Fix For: 1.14.0 Disk cost for Hbase scan is calculated based on scan size in bytes. {noformat} float diskCost = scanSizeInBytes * ((columns == null || columns.isEmpty()) ? 1 : columns.size() / statsCalculator.getColsPerRow()); {noformat} Scan size is bytes is estimated using {{TableStatsCalculator}} with the help of sampling. When we estimate size for the first time (before applying filter push down), for sampling we use random rows. When estimating rows after filter push down, for sampling we use rows that qualify filter condition. It can happen that average row size can be higher after filter push down than before. Unfortunately since disk cost depends on these calculations, plan with filter push down can give higher cost then without it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6435) MappingSet is stateful, so it can't be shared between threads
[ https://issues.apache.org/jira/browse/DRILL-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487457#comment-16487457 ] Volodymyr Vysotskyi commented on DRILL-6435: [~vrozov], thanks for catching this issue, agreed that is should be non-static as it was done in he most record batch classes. > MappingSet is stateful, so it can't be shared between threads > - > > Key: DRILL-6435 > URL: https://issues.apache.org/jira/browse/DRILL-6435 > Project: Apache Drill > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > > There are several instances where static {{MappingSet}} instances are used > (for example {{NestedLoopJoinBatch}} and {{BaseSortWrapper}}). This causes > instance reuse across threads when queries are executed concurrently. As > {{MappingSet}} is a stateful class with visitor design pattern, such reuse > causes invalid state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[ https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487375#comment-16487375 ] ASF GitHub Bot commented on DRILL-5977: --- akumarb2010 commented on issue #1272: DRILL-5977: Filter Pushdown in Drill-Kafka plugin URL: https://github.com/apache/drill/pull/1272#issuecomment-391372423 @aravi5 Sorry for the delay in review and thanks for implementing this nice feature. Before starting the code review, I have few comments on push down design. 1. *KafkaMsgOffset* predicates has to be partition specific right? We should not be applying these predicates globally across the partitions. For example p1[startOffset=1000, endOffset=2000], p2[1500,5000], So in this case, always better to consider the offsets per partition. But in test cases, I am not partition specific predicates. 2. Good to see that you have considered multiple scenarios for *kafkaMsgTimestamp* predicates and the above point will also applicable for *kafkaMsgTimestamp* as well. We might need to consider per partition specific *kafkaMsgTimestamp* predicates. But this might cause the issues, as * offsetsForTimes* method is a blocking and it can block indefinitely if user provides wrong partition. Can you please clarify above two comments? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > predicate pushdown support kafkaMsgOffset > - > > Key: DRILL-5977 > URL: https://issues.apache.org/jira/browse/DRILL-5977 > Project: Apache Drill > Issue Type: Improvement >Reporter: B Anil Kumar >Assignee: Abhishek Ravi >Priority: Major > Fix For: 1.14.0 > > > As part of Kafka storage plugin review, below is the suggestion from Paul. > {noformat} > Does it make sense to provide a way to select a range of messages: a starting > point or a count? Perhaps I want to run my query every five minutes, scanning > only those messages since the previous scan. Or, I want to limit my take to, > say, the next 1000 messages. Could we use a pseudo-column such as > "kafkaMsgOffset" for that purpose? Maybe > SELECT * FROM WHERE kafkaMsgOffset > 12345 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6425) Upgrade mapr release version
[ https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated DRILL-6425: -- Summary: Upgrade mapr release version (was: Upgrade org.ojai:ojai version) > Upgrade mapr release version > > > Key: DRILL-6425 > URL: https://issues.apache.org/jira/browse/DRILL-6425 > Project: Apache Drill > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6425) Upgrade mapr release version
[ https://issues.apache.org/jira/browse/DRILL-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated DRILL-6425: -- Description: Upgrade MapR dependendencies from {{5.2.1}} to {{6.0.1}}. (was: Upgrade from {{1.1}} to {{2.0.1-mapr-1804}} or the most recent.) > Upgrade mapr release version > > > Key: DRILL-6425 > URL: https://issues.apache.org/jira/browse/DRILL-6425 > Project: Apache Drill > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > Upgrade MapR dependendencies from {{5.2.1}} to {{6.0.1}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)