[jira] [Closed] (DRILL-6812) Release resources after finishing BloomFilterTest tests
[
https://issues.apache.org/jira/browse/DRILL-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weijie.tong closed DRILL-6812.
--
Resolution: Fixed
has fixed at DRILL-6792.
> Release resources after finishing BloomFilterTest tests
> ---
>
> Key: DRILL-6812
> URL: https://issues.apache.org/jira/browse/DRILL-6812
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Volodymyr Vysotskyi
>Assignee: weijie.tong
>Priority: Minor
>
> In every unit test from {{BloomFilterTest}} new {{Drillbit}} and
> {{FragmentContextImpl}} instances are created without closing them after the
> finishing tests and without clearing {{VectorContainer}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6784) Add RuntimeFilter metrics to the web console
[ https://issues.apache.org/jira/browse/DRILL-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weijie.tong closed DRILL-6784. -- Resolution: Fixed has fixed at DRILL-6792. > Add RuntimeFilter metrics to the web console > > > Key: DRILL-6784 > URL: https://issues.apache.org/jira/browse/DRILL-6784 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Monitoring, Web Server >Affects Versions: 1.15.0 >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Minor > > This issue is to add some RuntimeFilter metrics to the web console to watch > the RuntimeFilter behavior directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6727) JPPD does not eliminate rows using the bloom filter if a HashJoin is involved
[
https://issues.apache.org/jira/browse/DRILL-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weijie.tong closed DRILL-6727.
--
Resolution: Fixed
has fixed at DRILL-6792.
> JPPD does not eliminate rows using the bloom filter if a HashJoin is involved
> -
>
> Key: DRILL-6727
> URL: https://issues.apache.org/jira/browse/DRILL-6727
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Affects Versions: 1.15.0
>Reporter: Kunal Khatua
>Assignee: weijie.tong
>Priority: Critical
> Attachments:
> bcastJoin-JPPD_2477fb99-36cb-9bc2-b7fb-c81a52b256d2.json,
> bcastJoin-default_2477fa68-a31e-3b97-5469-373845c2b763.json,
> hashJoin-JPPD_2477f6f7-14e0-ca23-d9f7-6b0273c20964.json,
> hashJoin-default_2477f5e8-fff2-fc83-d251-d8be8f92820b.json
>
>
> When testing a simple join between 2 tables, it appears that the Bloom-filter
> based predicate pushdown will work only for broadcast joins, but not for
> hash-based joins.
> Since the purpose of the filter is to reduce the number of records being
> hashed across the fragments, the runtime does not improve.
> Join Query (TPCH dataset):
> {code:sql}
> select
> l.l_orderkey
> , sum(l.l_extendedprice * (1 - l.l_discount)) as revenue
> , o.o_orderdate
> , o.o_shippriority
> from
> orders o
> , lineitem l
> where
> l.l_orderkey = o.o_orderkey
> and o.o_orderdate = date '1994-08-26'
> and MOD(o.o_custkey,10) = 1
> group by
> l.l_orderkey
> , o.o_orderdate
> , o.o_shippriority
> order by
> revenue desc
> , o.o_orderdate limit 10;
> {code}
> This generates an output of about 6K rows from the build side, with the
> expectation of 10M rows being joined from the probe side.
> Following are the results of the following query:
> || Join Mode || Profile || Runtime || Status ||
> |BCastJoin w/o JPPD |
> [^bcastJoin-default_2477fa68-a31e-3b97-5469-373845c2b763.json] | 3.148sec |
> As expected. 600M rows are scanned and probed against the locally available
> hash table. |
> |BCastJoin w/ JPPD |
> [^bcastJoin-JPPD_2477fb99-36cb-9bc2-b7fb-c81a52b256d2.json] | 3.570sec |
> 04-xx-06 shows a reduction in rows. 600M rows are scanned, but only 10M rows
> are probed against the locally available hash table. |
> |
> |HashJoin w/o JPPD |
> [^hashJoin-default_2477f5e8-fff2-fc83-d251-d8be8f92820b.json] | 5.861sec |
> As expected. 600M rows are scanned and probed against the hash table. |
> |HashJoin w/ JPPD |
> [^hashJoin-JPPD_2477f6f7-14e0-ca23-d9f7-6b0273c20964.json] | 8.376sec |
> 03-xx-07 is not seeing a reduction in rows. All 600M rows are scanned and
> probed against the hash table. |
> There are a few possibilities of why the RuntimeFilter does not eliminate any
> rows when a HashJoin is involved.
> 1. The RuntimeFilter operator does not have a bloom-filter
> 2. The RuntimeFilter receives the bloom-filter after the scan completes,
> because the foreman has not finished building and distributing the global
> bloom-filter
> 3. The RuntimeFilter receives the bloom-filter during the scan, but does not
> apply it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DRILL-6718) TPCH Queries hit IOB when running with the JPPD feature
[
https://issues.apache.org/jira/browse/DRILL-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weijie.tong closed DRILL-6718.
--
Resolution: Fixed
has fixed at DRILL-6792
> TPCH Queries hit IOB when running with the JPPD feature
> ---
>
> Key: DRILL-6718
> URL: https://issues.apache.org/jira/browse/DRILL-6718
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.15.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: weijie.tong
>Priority: Major
> Attachments: TPCH_03_1_id_247b6925-b3f6-381f-12d2-7971abf317b7.json
>
>
> Running TPCH SF100 queries on drill master commit
> b895b28182a981e5948ffa292da827cb8b2e571e (DRILL-6385: Support JPPD feature).
> 9 queries hit IOB error, the stacks are similar, as follows
> {code}
> 2018-08-27 17:27:14,916 [247b6925-b3f6-381f-12d2-7971abf317b7:frag:5:35]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0], index: 0, length: 1
> (expected: range(0, 0))
> Fragment 5:35
> [Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0], index: 0, length: 1
> (expected: range(0, 0))
> Fragment 5:35
> [Error Id: a3de08b2-6453-4354-8d58-8a79d4c62469 on perf109-68.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:360)
> [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:215)
> [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:326)
> [drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [na:1.8.0_171]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [na:1.8.0_171]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
> Caused by: java.lang.IndexOutOfBoundsException: DrillBuf[1], udle: [1 0..0],
> index: 0, length: 1 (expected: range(0, 0))
> at
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:119)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
> at io.netty.buffer.DrillBuf.getByte(DrillBuf.java:757)
> ~[drill-memory-base-1.15.0-SNAPSHOT.jar:4.0.48.Final]
> at
> org.apache.drill.exec.vector.UInt1Vector$Accessor.get(UInt1Vector.java:424)
> ~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableBigIntVector$Accessor.isSet(NullableBigIntVector.java:446)
> ~[vector-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.Hash64Gen2282971.hash64Code(Hash64Gen2282971.java:62)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.computeBitSet(RuntimeFilterRecordBatch.java:241)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.applyRuntimeFilter(RuntimeFilterRecordBatch.java:219)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.filter.RuntimeFilterRecordBatch.doWork(RuntimeFilterRecordBatch.java:92)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.15.0-SNAPSHOT.jar:1.15.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
>
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683347#comment-16683347
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r232558044
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/RuntimeFilterVisitor.java
##
@@ -100,8 +102,18 @@ public Prel visitJoin(JoinPrel prel, Void value) throws
RuntimeException {
@Override
public Prel visitScan(ScanPrel prel, Void value) throws RuntimeException {
if (toAddRuntimeFilter.contains(prel)) {
- //Spawn a fresh RuntimeFilterPrel over the previous identified probe
side scan node.
- RuntimeFilterPrel runtimeFilterPrel = new RuntimeFilterPrel(prel);
+ //Spawn a fresh RuntimeFilterPrel over the previous identified probe
side scan node or a runtime filter node.
+ Collection hashJoinPrels = probeSideScan2hj.get(prel);
+ RuntimeFilterPrel runtimeFilterPrel = null;
+ for (HashJoinPrel hashJoinPrel : hashJoinPrels) {
+long identifier = atomicLong.incrementAndGet();
+
hashJoinPrel.getRuntimeFilterDef().setRuntimeFilterIdentifier(identifier);
+if (runtimeFilterPrel == null) {
+ runtimeFilterPrel = new RuntimeFilterPrel(prel, identifier);
+} else {
+ runtimeFilterPrel = new RuntimeFilterPrel(runtimeFilterPrel,
identifier);
Review comment:
Different RuntimeFilters may have different bytebuf size, the bits of the
bytebuf also have different meaning as the bucket chosen is different, so they
can't be aggregated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683304#comment-16683304
]
ASF GitHub Bot commented on DRILL-6792:
---
weijietong commented on a change in pull request #1504: DRILL-6792: Find the
right probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r232549977
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterReporter.java
##
@@ -58,12 +60,15 @@ public void sendOut(List bloomFilters,
List probeFields, bo
for (String probeFiled : probeFields) {
builder.addProbeFields(probeFiled);
}
+boolean comingFromHj = sendToForeman ? false : true;
Review comment:
That field has no help to the memory leak solution. It has been removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683302#comment-16683302
]
weijie.tong commented on DRILL-6838:
[~aravi5] I have reproduced the error and fixed that at the latest commit. The
actual reason is the close sequence of memory resource at the
FragmentContextImpl close method. The runtime filter should be closed before
the allocator closing.
Another enhancement is to release the runtime filter one more time if it
arrived at the probe side fragment context, but finally not consumed by the
RuntimeFilterRecordBatch.
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6791) Merge scan projection framework into master
[ https://issues.apache.org/jira/browse/DRILL-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683279#comment-16683279 ] ASF GitHub Bot commented on DRILL-6791: --- paul-rogers commented on issue #1501: DRILL-6791: Scan projection framework URL: https://github.com/apache/drill/pull/1501#issuecomment-437770579 @arina-ielchiieva, anything I should take care of on this one? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Merge scan projection framework into master > --- > > Key: DRILL-6791 > URL: https://issues.apache.org/jira/browse/DRILL-6791 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.15.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Major > Fix For: 1.15.0 > > > Merge the next set of "result set loader" code into master via a PR. This one > covers the "schema projection" mechanism which: > * Handles none (SELECT COUNT\(*)), some (SELECT a, b, x) and all (SELECT *) > projection. > * Handles null columns (for projection a column "x" that does not exist in > the base table.) > * Handles constant columns as used for file metadata (AKA "implicit" columns). > * Handle schema persistence: the need to reuse the same vectors across > different scanners > * Provides a framework for consuming externally-supplied metadata > * Since we don't yet have a way to provide "real" metadata, obtains metadata > hints from previous batches and from the projection list (a.b implies that > "a" is a map, c[0] implies that "c" is an array, etc.) > * Handles merging the set of data source columns and null columns to create > the final output batch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683278#comment-16683278
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on issue #1531: DRILL-5571: Cancel running query from its Web
UI
URL: https://github.com/apache/drill/pull/1531#issuecomment-437770292
@arina-ielchiieva / @sohami made changes based on review comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683261#comment-16683261
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232543607
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -171,7 +229,17 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#assign queued = queueName != "" && queueName != "-" />
- Query Profile : ${model.getQueryId()}
+ Query Profile : ${model.getQueryId()}
+ <#if model.getQueryStateDisplayName() == "Running" ||
model.getQueryStateDisplayName() == "Planning">
+
+Issued cancellation for queryClick
to continue
+
+
+
+ Cancel
Review comment:
Ok. Dropped the glyphicon and retained the warning (orange) cancellation
button. I didn't want it to be a standard blue button as this is a query
termination button.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683259#comment-16683259
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232543357
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -50,6 +50,34 @@
"info": false
}
);} );
+
+//Show cancellation status
+function showCancelPopup(queryId) {
+document.getElementById("cancelText").innerHTML = "Issued cancellation
for query ID: "+queryId+"Click to continue";
+document.getElementById("cancelPopup").style.display = "block";
+}
+
+//Close the cancellation status popup
+function closePopup() {
+document.getElementById("cancelPopup").style.display = "none";
+location.reload(true);
+}
+
+//Cancel query
+function cancelQuery(queryId) {
+var cancelRestURL = location.protocol + "//" + location.hostname +
"/profiles/cancel/" + queryId;
+let restPort = location.port;
+if (restPort != 0) {
Review comment:
`location.port` returns 0 if the port number is standard. However, I've
changed this because I realized I don't really need the server as the
`/profiles/cancel/...` will be made on the same server that served the profile
UI.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683212#comment-16683212
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r232529229
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/RuntimeFilterVisitor.java
##
@@ -100,8 +102,18 @@ public Prel visitJoin(JoinPrel prel, Void value) throws
RuntimeException {
@Override
public Prel visitScan(ScanPrel prel, Void value) throws RuntimeException {
if (toAddRuntimeFilter.contains(prel)) {
- //Spawn a fresh RuntimeFilterPrel over the previous identified probe
side scan node.
- RuntimeFilterPrel runtimeFilterPrel = new RuntimeFilterPrel(prel);
+ //Spawn a fresh RuntimeFilterPrel over the previous identified probe
side scan node or a runtime filter node.
+ Collection hashJoinPrels = probeSideScan2hj.get(prel);
+ RuntimeFilterPrel runtimeFilterPrel = null;
+ for (HashJoinPrel hashJoinPrel : hashJoinPrels) {
+long identifier = atomicLong.incrementAndGet();
+
hashJoinPrel.getRuntimeFilterDef().setRuntimeFilterIdentifier(identifier);
+if (runtimeFilterPrel == null) {
+ runtimeFilterPrel = new RuntimeFilterPrel(prel, identifier);
+} else {
+ runtimeFilterPrel = new RuntimeFilterPrel(runtimeFilterPrel,
identifier);
Review comment:
in this case it will generate 2 RuntimeFilter's stacked on top of each
other. Shouldn't we instead combine the 2 RTF into single RTF with bloomFilter
aggregation using some AND operation ? Example of case where this may happen:
HJ
/ \
HJ Scan
/ \
Scan Scan
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683214#comment-16683214 ] ASF GitHub Bot commented on DRILL-6792: --- sohami commented on a change in pull request #1504: DRILL-6792: Find the right probe side fragment wrapper & fix DrillBuf… URL: https://github.com/apache/drill/pull/1504#discussion_r232526727 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/RuntimeFilterVisitor.java ## @@ -58,9 +55,14 @@ private Set toAddRuntimeFilter = new HashSet<>(); + private Multimap probeSideScan2hj = HashMultimap.create(); + private double fpp; + private int bloomFilterMaxSizeInBytesDef; + private static final AtomicLong atomicLong = new AtomicLong(); Review comment: please rename this variable to something like `rtfIdCounter`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Find the right probe side fragment to any storage plugin > > > Key: DRILL-6792 > URL: https://issues.apache.org/jira/browse/DRILL-6792 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Flow >Reporter: weijie.tong >Assignee: weijie.tong >Priority: Major > Fix For: 1.15.0 > > > The current implementation of JPPD to find the probe side wrapper depends on > the GroupScan's digest. But there's no promise the GroupScan's digest will > not be changed since it is attached to the RuntimeFilterDef by different > storage plugin implementation logic.So here we assign a unique identifier to > the RuntimeFilter operator, and find the right probe side fragment wrapper by > the runtime filter identifier at the RuntimeFilterRouter class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6792) Find the right probe side fragment to any storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683213#comment-16683213
]
ASF GitHub Bot commented on DRILL-6792:
---
sohami commented on a change in pull request #1504: DRILL-6792: Find the right
probe side fragment wrapper & fix DrillBuf…
URL: https://github.com/apache/drill/pull/1504#discussion_r232526049
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/filter/RuntimeFilterReporter.java
##
@@ -58,12 +60,15 @@ public void sendOut(List bloomFilters,
List probeFields, bo
for (String probeFiled : probeFields) {
builder.addProbeFields(probeFiled);
}
+boolean comingFromHj = sendToForeman ? false : true;
Review comment:
It looks like `comingFromHj` is basically `!sendToForeman` and
`sendToForeman` is already been set in `toForeman` field of
`RuntimeFilterBDef`. So `comingFromHj` field is redundant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Find the right probe side fragment to any storage plugin
>
>
> Key: DRILL-6792
> URL: https://issues.apache.org/jira/browse/DRILL-6792
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
>Reporter: weijie.tong
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
>
> The current implementation of JPPD to find the probe side wrapper depends on
> the GroupScan's digest. But there's no promise the GroupScan's digest will
> not be changed since it is attached to the RuntimeFilterDef by different
> storage plugin implementation logic.So here we assign a unique identifier to
> the RuntimeFilter operator, and find the right probe side fragment wrapper by
> the runtime filter identifier at the RuntimeFilterRouter class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683159#comment-16683159
]
weijie.tong commented on DRILL-6838:
Sorry ,yesterday is too late to verify the changed code. I will test and fix it
today.
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683134#comment-16683134
]
ASF GitHub Bot commented on DRILL-5571:
---
sohami commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232520154
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -50,6 +50,34 @@
"info": false
}
);} );
+
+//Show cancellation status
+function showCancelPopup(queryId) {
+document.getElementById("cancelText").innerHTML = "Issued cancellation
for query ID: "+queryId+"Click to continue";
+document.getElementById("cancelPopup").style.display = "block";
+}
+
+//Close the cancellation status popup
+function closePopup() {
+document.getElementById("cancelPopup").style.display = "none";
+location.reload(true);
+}
+
+//Cancel query
+function cancelQuery(queryId) {
+var cancelRestURL = location.protocol + "//" + location.hostname +
"/profiles/cancel/" + queryId;
+let restPort = location.port;
+if (restPort != 0) {
Review comment:
in what case `restPort` will be 0 ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683133#comment-16683133
]
ASF GitHub Bot commented on DRILL-5571:
---
sohami commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232521155
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -171,7 +229,17 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#assign queued = queueName != "" && queueName != "-" />
- Query Profile : ${model.getQueryId()}
+ Query Profile : ${model.getQueryId()}
+ <#if model.getQueryStateDisplayName() == "Running" ||
model.getQueryStateDisplayName() == "Planning">
+
+Issued cancellation for queryClick
to continue
+
+
+
+ Cancel
Review comment:
Not to be picky but I would say for consistency use the same button format
as in other pages like /storage tab.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683077#comment-16683077
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232511516
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -50,6 +50,34 @@
"info": false
}
);} );
+
+//Show cancellation status
+function showCancelPopup(queryId) {
Review comment:
Wasn't aware of this and not sure if the current Bootstrap supports this.
Will try and if not supported in the current version within Drill, let's
continue with the CSS.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683072#comment-16683072
]
Abhishek Ravi edited comment on DRILL-6838 at 11/11/18 11:56 PM:
-
[~weijie] - thanks for the changes but I am afraid the issue still exists even
after the latest changes.
h3. Execution Failures:
{noformat}
Query:
/root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683075#comment-16683075
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232511339
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -50,6 +50,34 @@
"info": false
}
);} );
+
+//Show cancellation status
+function showCancelPopup(queryId) {
+document.getElementById("cancelText").innerHTML = "Issued cancellation
for query ID: "+queryId+"Click to continue";
+document.getElementById("cancelPopup").style.display = "block";
+}
+
+//Close the cancellation status popup
+function closePopup() {
+document.getElementById("cancelPopup").style.display = "none";
+location.reload(true);
+}
+
+//Cancel query
+function cancelQuery(queryId) {
+var cancelRestURL = location.protocol + "//" + location.hostname +
"/profiles/cancel/" + queryId;
+let restPort = location.port;
+if (restPort != 0) {
+cancelRestURL = location.protocol + "//" + location.hostname + ":"
+ restPort + "/profiles/cancel/" + queryId;
+}
+$.get(cancelRestURL, function(data, status){
+//dBug
+console.log("Cancelled >--> " + cancelRestURL);
Review comment:
Missed this. Shouldn't matter, though, because the page refreshes after the
popup is closed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683072#comment-16683072
]
Abhishek Ravi commented on DRILL-6838:
--
[~weijie] - thanks for the changes but I am afraid the issue still exists even
after the latest changes.
h1. h3. Execution Failures:
{noformat}
Query:
/root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683074#comment-16683074
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232511312
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -171,7 +229,17 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#assign queued = queueName != "" && queueName != "-" />
- Query Profile : ${model.getQueryId()}
+ Query Profile : ${model.getQueryId()}
+ <#if model.getQueryStateDisplayName() == "Running" ||
model.getQueryStateDisplayName() == "Planning">
+
+Issued cancellation for queryClick
to continue
+
+
+
+ Cancel
Review comment:
Ok. I thought adding an icon makes it more visually informative.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6806) Start moving code for handling a partition in HashAgg into a separate class
[ https://issues.apache.org/jira/browse/DRILL-6806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6806: - Reviewer: Boaz Ben-Zvi > Start moving code for handling a partition in HashAgg into a separate class > --- > > Key: DRILL-6806 > URL: https://issues.apache.org/jira/browse/DRILL-6806 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Fix For: 1.15.0 > > > Since this involves a lot of refactoring this will be a multiple PR effort. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683072#comment-16683072
]
Abhishek Ravi edited comment on DRILL-6838 at 11/11/18 11:55 PM:
-
[~weijie] - thanks for the changes but I am afraid the issue still exists even
after the latest changes.
h3. Execution Failures:
{noformat}
Query:
/root/my-workspace/drill-test-framework/framework/resources/Advanced/runtimefilter/tpch_sf100/parquet/broadcast_hash_01.q
select
l.l_orderkey,
sum(l.l_extendedprice * (1 - l.l_discount)) as revenue,
o.o_orderdate,
o.o_shippriority
from
customer c,
orders o,
lineitem l
where
c.c_mktsegment = 'HOUSEHOLD'
and c.c_custkey = o.o_custkey
and l.l_orderkey = o.o_orderkey
and o.o_orderdate < date '1995-03-25'
and l.l_shipdate > date '1995-03-25'
group by
l.l_orderkey,
o.o_orderdate,
o.o_shippriority
order by
revenue desc,
o.o_orderdate
limit 10
Exception:
java.sql.SQLException: SYSTEM ERROR: IllegalStateException: Memory was leaked
by query. Memory leaked: (4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:528)
at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:632)
at
oadd.org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:217)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:151)
at
org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:253)
at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: IllegalStateException: Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
Fragment 3:41
Please, refer to logs for more information.
[Error Id: 717a8e4d-142d-4da2-b7a9-75c74904854d on qa-node185.qa.lab:31010]
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked:
(4194304)
Allocator(frag:3:41) 800/4194304/81947456/60421075224
(res/actual/peak/limit)
org.apache.drill.exec.memory.BaseAllocator.close():520
org.apache.drill.exec.ops.FragmentContextImpl.suppressingClose():544
org.apache.drill.exec.ops.FragmentContextImpl.close():537
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources():387
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup():214
org.apache.drill.exec.work.fragment.FragmentExecutor.run():330
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():748
at
oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683071#comment-16683071
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232511275
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -171,7 +229,17 @@ table.sortable thead .sorting_desc { background-image:
url("/static/img/black-de
<#assign queued = queueName != "" && queueName != "-" />
- Query Profile : ${model.getQueryId()}
+ Query Profile : ${model.getQueryId()}
Review comment:
Ok. I think it might have been there to make it more readable with the
actual query ID.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5571) Unable to cancel running queries from Web UI
[
https://issues.apache.org/jira/browse/DRILL-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683070#comment-16683070
]
ASF GitHub Bot commented on DRILL-5571:
---
kkhatua commented on a change in pull request #1531: DRILL-5571: Cancel running
query from its Web UI
URL: https://github.com/apache/drill/pull/1531#discussion_r232511213
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -50,6 +50,34 @@
"info": false
}
);} );
+
+//Show cancellation status
+function showCancelPopup(queryId) {
+document.getElementById("cancelText").innerHTML = "Issued cancellation
for query ID: "+queryId+"Click to continue";
+document.getElementById("cancelPopup").style.display = "block";
+}
+
+//Close the cancellation status popup
+function closePopup() {
+document.getElementById("cancelPopup").style.display = "none";
+location.reload(true);
+}
+
+//Cancel query
+function cancelQuery(queryId) {
+var cancelRestURL = location.protocol + "//" + location.hostname +
"/profiles/cancel/" + queryId;
Review comment:
My assumption is that if HTTPS is enabled, the protocol should reflect it,
though I probably just needed `"/profiles/cancel/" + queryId`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unable to cancel running queries from Web UI
>
>
> Key: DRILL-5571
> URL: https://issues.apache.org/jira/browse/DRILL-5571
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - HTTP
>Affects Versions: 1.11.0
>Reporter: Kedar Sankar Behera
>Assignee: Kunal Khatua
>Priority: Major
> Labels: javascript, newbie
> Fix For: 1.15.0
>
>
> We are unable to access profiles of some running queries. Hit the following
> error on the Web UI:
> {code}
> {
> “errorMessage” : “VALIDATION ERROR: No profile with given query id
> ‘26c90b95-928b-15e3-bedc-bfb4a046cc8b’ exists. Please verify the query
> id.\n\n\n[Error Id: e6896a23-6932-469d-9968-d315fdd06dd4 ]”
> }
> {code}
> And we cannot cancel the running queries whose profile page can be accessed:
> {code}
> Failure attempting to cancel query 26c90b33-cf7e-0495-8f76-55220f71f809.
> Unable to find information about where query is actively running.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6740) Missing ODBC Driver for v1.14
[ https://issues.apache.org/jira/browse/DRILL-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683063#comment-16683063 ] Kunal Khatua commented on DRILL-6740: - [~teroz] New Drivers have already been uploaded and Preview drivers have been removed to avoid confusion: Official ODBC Drivers for Apache Drill 1.14.0 are available here: https://package.mapr.com/tools/MapR-ODBC/MapR_Drill/MapRDrill_odbc_v1.3.22.1055/ Official JDBC Drivers (for Linux only) are available here: https://package.mapr.com/tools/MapR-JDBC/MapR_Drill/MapRDrill_jdbc_v1.6.0.1001/ > Missing ODBC Driver for v1.14 > - > > Key: DRILL-6740 > URL: https://issues.apache.org/jira/browse/DRILL-6740 > Project: Apache Drill > Issue Type: Bug > Components: Client - ODBC >Affects Versions: 1.14.0 > Environment: Ubuntu 16.04.5 LTS, 4.15.0-29-generic #31~16.04.1, x86_64 >Reporter: Terence Namusonge Sifuna >Assignee: Kunal Khatua >Priority: Blocker > Attachments: 276px-Unformat.gif > > > Unable to Use the current odbc driver with v1.14 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6830) Hook.REL_BUILDER_SIMPLIFY handler didn't removed cause performance degression
[ https://issues.apache.org/jira/browse/DRILL-6830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6830: - Reviewer: Volodymyr Vysotskyi > Hook.REL_BUILDER_SIMPLIFY handler didn't removed cause performance degression > - > > Key: DRILL-6830 > URL: https://issues.apache.org/jira/browse/DRILL-6830 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning Optimization >Affects Versions: 1.14.0 >Reporter: shuifeng lu >Assignee: shuifeng lu >Priority: Major > Fix For: 1.15.0 > > Attachments: Screen Shot 2018-11-06 at 16.14.16.png > > > Planning performance degression has been observed that the duration of > planning increased from 30ms to 160ms after running drill a long period of > time(say a month). > RelBuilder.simplify never becomes true if Hook.REL_BUILDER_SIMPLIFY handlers > are not removed. > Here is some clue (after running 40 days): > Hook.get takes 8ms per-invocation, it may be called serveral times per query. > ---[8.816063ms] org.apache.calcite.tools.RelBuilder:() > +---[0.020218ms] java.util.ArrayDeque:() > +---[0.018493ms] java.lang.Boolean:valueOf() > +---[8.341566ms] org.apache.calcite.runtime.Hook:get() > +---[0.008489ms] java.lang.Boolean:booleanValue() > +---[min=5.21E-4ms,max=0.015832ms,total=0.025233ms,count=12] > org.apache.calcite.plan.Context:unwrap() > +---[min=3.83E-4ms,max=0.009494ms,total=0.014516ms,count=13] > org.apache.calcite.util.Util:first() > +---[0.006892ms] org.apache.calcite.plan.RelOptCluster:getPlanner() > +---[0.009104ms] org.apache.calcite.plan.RelOptPlanner:getExecutor() > +---[min=4.8E-4ms,max=0.002277ms,total=0.002757ms,count=2] > org.apache.calcite.plan.RelOptCluster:getRexBuilder() > ---[min=4.91E-4ms,max=0.004586ms,total=0.005077ms,count=2] > org.apache.calcite.rex.RexSimplify:() > The top instances in JVM > num #instances #bytes class name > -- > 1: 116333 116250440 [B > 2: 890126 105084536 [C > 3: 338062 37415944 [Ljava.lang.Object; > 4: 1715004 27440064 org.apache.calcite.runtime.Hook$4 > 5: 803909 19293816 java.lang.String > !Screen Shot 2018-11-06 at 16.14.16.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6740) Missing ODBC Driver for v1.14
[ https://issues.apache.org/jira/browse/DRILL-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua closed DRILL-6740. --- > Missing ODBC Driver for v1.14 > - > > Key: DRILL-6740 > URL: https://issues.apache.org/jira/browse/DRILL-6740 > Project: Apache Drill > Issue Type: Bug > Components: Client - ODBC >Affects Versions: 1.14.0 > Environment: Ubuntu 16.04.5 LTS, 4.15.0-29-generic #31~16.04.1, x86_64 >Reporter: Terence Namusonge Sifuna >Assignee: Kunal Khatua >Priority: Blocker > Attachments: 276px-Unformat.gif > > > Unable to Use the current odbc driver with v1.14 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6844) Query with ORDER BY DESC on indexed column does not pick secondary index
[
https://issues.apache.org/jira/browse/DRILL-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6844:
-
Reviewer: Aman Sinha
> Query with ORDER BY DESC on indexed column does not pick secondary index
>
>
> Key: DRILL-6844
> URL: https://issues.apache.org/jira/browse/DRILL-6844
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
> Query with ORDER BY DESC on indexed column does not pick secondary index
> {noformat}
> // Query that uses the secondary index defined on ts.
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select ts from dfs.`/c8/test3` order by ts
> limit 1;
> +--+--+
> | text | json |
> +--+--+
> | 00-00 Screen
> 00-01 Project(ts=[$0])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[1])
> 00-04 Scan(table=[[dfs, /c8/test3]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///c8/test3, condition=null,
> indexName=ts], columns=[`ts`], limit=1, maxwidth=125]])
> {noformat}
> // Same query with ORDER BY ts DESC does not use the secondary index defined
> on ts.
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select ts from dfs.`/c8/test3` order by ts desc
> limit 1;
> +--+--+
> | text | json |
> +--+--+
> | 00-00 Screen
> 00-01 Project(ts=[$0])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[1])
> 00-04 SingleMergeExchange(sort0=[0 DESC])
> 01-01 OrderedMuxExchange(sort0=[0 DESC])
> 02-01 SelectionVectorRemover
> 02-02 Limit(fetch=[1])
> 02-03 SelectionVectorRemover
> 02-04 TopN(limit=[1])
> 02-05 HashToRandomExchange(dist0=[[$0]])
> 03-01 Scan(table=[[dfs, /c8/test3]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///c8/test3, condition=null],
> columns=[`ts`], maxwidth=8554]])
> {noformat}
> { noformat}
> Index definition is,
> maprcli table index list -path /c8/test3 -json
> {
> "timestamp":1538066303932,
> "timeofday":"2018-09-27 04:38:23.932 GMT+ PM",
> "status":"OK",
> "total":2,
> "data":[
> {
> "cluster":"c8",
> "type":"maprdb.si",
> "indexFid":"2176.68.131294",
> "indexName":"ts",
> "hashed":false,
> "indexState":"REPLICA_STATE_REPLICATING",
> "idx":1,
> "indexedFields":"ts:ASC",
> "isUptodate":false,
> "minPendingTS":1538066077,
> "maxPendingTS":1538066077,
> "bytesPending":0,
> "putsPending":0,
> "bucketsPending":1,
> "copyTableCompletionPercentage":100,
> "numTablets":32,
> "numRows":80574368,
> "totalSize":4854052160
> },
> {
> "cluster":"c8",
> "type":"maprdb.si",
> "indexFid":"2176.72.131302",
> "indexName":"ts_desc",
> "hashed":false,
> "indexState":"REPLICA_STATE_REPLICATING",
> "idx":2,
> "indexedFields":"ts:DESC",
> "isUptodate":false,
> "minPendingTS":1538066077,
> "maxPendingTS":1538066077,
> "bytesPending":0,
> "putsPending":0,
> "bucketsPending":1,
> "copyTableCompletionPercentage":100,
> "numTablets":32,
> "numRows":80081344,
> "totalSize":4937154560
> }
> ]
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6844) Query with ORDER BY DESC on indexed column does not pick secondary index
[
https://issues.apache.org/jira/browse/DRILL-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6844:
-
Fix Version/s: 1.15.0
> Query with ORDER BY DESC on indexed column does not pick secondary index
>
>
> Key: DRILL-6844
> URL: https://issues.apache.org/jira/browse/DRILL-6844
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
> Fix For: 1.15.0
>
>
> Query with ORDER BY DESC on indexed column does not pick secondary index
> {noformat}
> // Query that uses the secondary index defined on ts.
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select ts from dfs.`/c8/test3` order by ts
> limit 1;
> +--+--+
> | text | json |
> +--+--+
> | 00-00 Screen
> 00-01 Project(ts=[$0])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[1])
> 00-04 Scan(table=[[dfs, /c8/test3]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///c8/test3, condition=null,
> indexName=ts], columns=[`ts`], limit=1, maxwidth=125]])
> {noformat}
> // Same query with ORDER BY ts DESC does not use the secondary index defined
> on ts.
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select ts from dfs.`/c8/test3` order by ts desc
> limit 1;
> +--+--+
> | text | json |
> +--+--+
> | 00-00 Screen
> 00-01 Project(ts=[$0])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[1])
> 00-04 SingleMergeExchange(sort0=[0 DESC])
> 01-01 OrderedMuxExchange(sort0=[0 DESC])
> 02-01 SelectionVectorRemover
> 02-02 Limit(fetch=[1])
> 02-03 SelectionVectorRemover
> 02-04 TopN(limit=[1])
> 02-05 HashToRandomExchange(dist0=[[$0]])
> 03-01 Scan(table=[[dfs, /c8/test3]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///c8/test3, condition=null],
> columns=[`ts`], maxwidth=8554]])
> {noformat}
> { noformat}
> Index definition is,
> maprcli table index list -path /c8/test3 -json
> {
> "timestamp":1538066303932,
> "timeofday":"2018-09-27 04:38:23.932 GMT+ PM",
> "status":"OK",
> "total":2,
> "data":[
> {
> "cluster":"c8",
> "type":"maprdb.si",
> "indexFid":"2176.68.131294",
> "indexName":"ts",
> "hashed":false,
> "indexState":"REPLICA_STATE_REPLICATING",
> "idx":1,
> "indexedFields":"ts:ASC",
> "isUptodate":false,
> "minPendingTS":1538066077,
> "maxPendingTS":1538066077,
> "bytesPending":0,
> "putsPending":0,
> "bucketsPending":1,
> "copyTableCompletionPercentage":100,
> "numTablets":32,
> "numRows":80574368,
> "totalSize":4854052160
> },
> {
> "cluster":"c8",
> "type":"maprdb.si",
> "indexFid":"2176.72.131302",
> "indexName":"ts_desc",
> "hashed":false,
> "indexState":"REPLICA_STATE_REPLICATING",
> "idx":2,
> "indexedFields":"ts:DESC",
> "isUptodate":false,
> "minPendingTS":1538066077,
> "maxPendingTS":1538066077,
> "bytesPending":0,
> "putsPending":0,
> "bucketsPending":1,
> "copyTableCompletionPercentage":100,
> "numTablets":32,
> "numRows":80081344,
> "totalSize":4937154560
> }
> ]
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6841) Statistics ANALYZE statement should be more flexible
[ https://issues.apache.org/jira/browse/DRILL-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker reassigned DRILL-6841: Assignee: Gautam Parai > Statistics ANALYZE statement should be more flexible > > > Key: DRILL-6841 > URL: https://issues.apache.org/jira/browse/DRILL-6841 > Project: Apache Drill > Issue Type: Improvement >Reporter: Gautam Parai >Assignee: Gautam Parai >Priority: Major > > Investigate and implement other alternatives to the ANALYZE statement e.g. > SELECT ndv(id) FROM OR enhance the ANALYZE statement to support such > options. This may be helpful for users to experiment easily and quickly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6839) Failed to plan (aggregate + Hash or NL join) when slice target is low
[
https://issues.apache.org/jira/browse/DRILL-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6839:
-
Fix Version/s: 1.16.0
> Failed to plan (aggregate + Hash or NL join) when slice target is low
> --
>
> Key: DRILL-6839
> URL: https://issues.apache.org/jira/browse/DRILL-6839
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Igor Guzenko
>Priority: Major
> Fix For: 1.16.0
>
>
> *Case 1.* When nested loop join is about to be used:
> - Option "_planner.enable_nljoin_for_scalar_only_" is set to false
> - Option "_planner.slice_target_" is set to low value for imitation of big
> input tables
>
> {code:java}
> @Category(SqlTest.class)
> public class CrossJoinTest extends ClusterTest {
> @BeforeClass
> public static void setUp() throws Exception {
> startCluster(ClusterFixture.builder(dirTestWatcher));
> }
> @Test
> public void testCrossJoinSucceedsForLowSliceTarget() throws Exception {
>try {
> client.alterSession(PlannerSettings.NLJOIN_FOR_SCALAR.getOptionName(),
> false);
> client.alterSession(ExecConstants.SLICE_TARGET, 1);
> queryBuilder().sql(
> "SELECT COUNT(l.nation_id) " +
> "FROM cp.`tpch/nation.parquet` l " +
> ", cp.`tpch/region.parquet` r")
> .run();
>} finally {
> client.resetSession(ExecConstants.SLICE_TARGET);
> client.resetSession(PlannerSettings.NLJOIN_FOR_SCALAR.getOptionName());
>}
> }
> }{code}
>
> *Case 2.* When hash join is about to be used:
> - Option "planner.enable_mergejoin" is set to false, so hash join will be
> used instead
> - Option "planner.slice_target" is set to low value for imitation of big
> input tables
> - Comment out //ruleList.add(HashJoinPrule.DIST_INSTANCE); in
> PlannerPhase.getPhysicalRules method
> {code:java}
> @Category(SqlTest.class)
> public class CrossJoinTest extends ClusterTest {
> @BeforeClass
> public static void setUp() throws Exception {
>startCluster(ClusterFixture.builder(dirTestWatcher));
> }
> @Test
> public void testInnerJoinSucceedsForLowSliceTarget() throws Exception {
>try {
> client.alterSession(PlannerSettings.MERGEJOIN.getOptionName(), false);
> client.alterSession(ExecConstants.SLICE_TARGET, 1);
> queryBuilder().sql(
> "SELECT COUNT(l.nation_id) " +
> "FROM cp.`tpch/nation.parquet` l " +
> "INNER JOIN cp.`tpch/region.parquet` r " +
> "ON r.nation_id = l.nation_id")
> .run();
>} finally {
> client.resetSession(ExecConstants.SLICE_TARGET);
> client.resetSession(PlannerSettings.MERGEJOIN.getOptionName());
>}
> }
> }
> {code}
>
> *Workaround:* To avoid the exception we need to set option
> "_planner.enable_multiphase_agg_" to false. By doing this we avoid
> unsuccessful attempts to create 2 phase aggregation plan in StreamAggPrule
> and guarantee that logical aggregate will be converted to physical one.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683061#comment-16683061
]
Pritesh Maker commented on DRILL-6838:
--
thanks [~weijie] . [~aravi5] do verify it and mark the issue resolved.
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6838:
-
Fix Version/s: 1.15.0
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6835) Schema Provision using File / Table Function
[ https://issues.apache.org/jira/browse/DRILL-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6835: - Fix Version/s: 1.16.0 > Schema Provision using File / Table Function > > > Key: DRILL-6835 > URL: https://issues.apache.org/jira/browse/DRILL-6835 > Project: Apache Drill > Issue Type: New Feature >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva >Priority: Major > Fix For: 1.16.0 > > > Schema Provision using File / Table Function design document: > https://docs.google.com/document/d/1mp4egSbNs8jFYRbPVbm_l0Y5GjH3HnoqCmOpMTR_g4w/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6831) Adding information about possible authentication settings in connection URL for mongodb
[ https://issues.apache.org/jira/browse/DRILL-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker reassigned DRILL-6831: Assignee: Gabriel Filipiak > Adding information about possible authentication settings in connection URL > for mongodb > --- > > Key: DRILL-6831 > URL: https://issues.apache.org/jira/browse/DRILL-6831 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Reporter: Gabriel Filipiak >Assignee: Gabriel Filipiak >Priority: Major > Fix For: 1.15.0 > > > Adding information about possible authentication settings in connection URL > for mongodb -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6831) Adding information about possible authentication settings in connection URL for mongodb
[ https://issues.apache.org/jira/browse/DRILL-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6831: - Fix Version/s: 1.15.0 > Adding information about possible authentication settings in connection URL > for mongodb > --- > > Key: DRILL-6831 > URL: https://issues.apache.org/jira/browse/DRILL-6831 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Reporter: Gabriel Filipiak >Priority: Major > Fix For: 1.15.0 > > > Adding information about possible authentication settings in connection URL > for mongodb -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6831) Adding information about possible authentication settings in connection URL for mongodb
[ https://issues.apache.org/jira/browse/DRILL-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683058#comment-16683058 ] ASF GitHub Bot commented on DRILL-6831: --- priteshm commented on issue #1525: DRILL-6831: Adding information about possible authentication settings in connection URL for mongodb URL: https://github.com/apache/drill/pull/1525#issuecomment-437716359 @bbevens could you please complete the review and merge to the docs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Adding information about possible authentication settings in connection URL > for mongodb > --- > > Key: DRILL-6831 > URL: https://issues.apache.org/jira/browse/DRILL-6831 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Reporter: Gabriel Filipiak >Priority: Major > Fix For: 1.15.0 > > > Adding information about possible authentication settings in connection URL > for mongodb -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6831) Adding information about possible authentication settings in connection URL for mongodb
[ https://issues.apache.org/jira/browse/DRILL-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6831: - Issue Type: Task (was: Improvement) > Adding information about possible authentication settings in connection URL > for mongodb > --- > > Key: DRILL-6831 > URL: https://issues.apache.org/jira/browse/DRILL-6831 > Project: Apache Drill > Issue Type: Task > Components: Documentation >Reporter: Gabriel Filipiak >Priority: Major > > Adding information about possible authentication settings in connection URL > for mongodb -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6844) Query with ORDER BY DESC on indexed column does not pick secondary index
[
https://issues.apache.org/jira/browse/DRILL-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683055#comment-16683055
]
ASF GitHub Bot commented on DRILL-6844:
---
HanumathRao opened a new pull request #1534: DRILL-6844: Query with ORDER BY
DESC on indexed column does not pick …
URL: https://github.com/apache/drill/pull/1534
…secondary index.
This PR fixes the issue related to query not picking index when desc is used
in the order by clause.
The code is changed to create the sort collation with nulls first when
indexed field is desc otherwise it creates nulls last.
@amansinha100 Can you please review this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query with ORDER BY DESC on indexed column does not pick secondary index
>
>
> Key: DRILL-6844
> URL: https://issues.apache.org/jira/browse/DRILL-6844
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning Optimization
>Affects Versions: 1.14.0
>Reporter: Hanumath Rao Maduri
>Assignee: Hanumath Rao Maduri
>Priority: Major
>
> Query with ORDER BY DESC on indexed column does not pick secondary index
> {noformat}
> // Query that uses the secondary index defined on ts.
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select ts from dfs.`/c8/test3` order by ts
> limit 1;
> +--+--+
> | text | json |
> +--+--+
> | 00-00 Screen
> 00-01 Project(ts=[$0])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[1])
> 00-04 Scan(table=[[dfs, /c8/test3]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///c8/test3, condition=null,
> indexName=ts], columns=[`ts`], limit=1, maxwidth=125]])
> {noformat}
> // Same query with ORDER BY ts DESC does not use the secondary index defined
> on ts.
> 0: jdbc:drill:schema=dfs.tmp> explain plan for
> . . . . . . . . . . . . . . > select ts from dfs.`/c8/test3` order by ts desc
> limit 1;
> +--+--+
> | text | json |
> +--+--+
> | 00-00 Screen
> 00-01 Project(ts=[$0])
> 00-02 SelectionVectorRemover
> 00-03 Limit(fetch=[1])
> 00-04 SingleMergeExchange(sort0=[0 DESC])
> 01-01 OrderedMuxExchange(sort0=[0 DESC])
> 02-01 SelectionVectorRemover
> 02-02 Limit(fetch=[1])
> 02-03 SelectionVectorRemover
> 02-04 TopN(limit=[1])
> 02-05 HashToRandomExchange(dist0=[[$0]])
> 03-01 Scan(table=[[dfs, /c8/test3]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///c8/test3, condition=null],
> columns=[`ts`], maxwidth=8554]])
> {noformat}
> { noformat}
> Index definition is,
> maprcli table index list -path /c8/test3 -json
> {
> "timestamp":1538066303932,
> "timeofday":"2018-09-27 04:38:23.932 GMT+ PM",
> "status":"OK",
> "total":2,
> "data":[
> {
> "cluster":"c8",
> "type":"maprdb.si",
> "indexFid":"2176.68.131294",
> "indexName":"ts",
> "hashed":false,
> "indexState":"REPLICA_STATE_REPLICATING",
> "idx":1,
> "indexedFields":"ts:ASC",
> "isUptodate":false,
> "minPendingTS":1538066077,
> "maxPendingTS":1538066077,
> "bytesPending":0,
> "putsPending":0,
> "bucketsPending":1,
> "copyTableCompletionPercentage":100,
> "numTablets":32,
> "numRows":80574368,
> "totalSize":4854052160
> },
> {
> "cluster":"c8",
> "type":"maprdb.si",
> "indexFid":"2176.72.131302",
> "indexName":"ts_desc",
> "hashed":false,
> "indexState":"REPLICA_STATE_REPLICATING",
> "idx":2,
> "indexedFields":"ts:DESC",
> "isUptodate":false,
> "minPendingTS":1538066077,
> "maxPendingTS":1538066077,
> "bytesPending":0,
> "putsPending":0,
> "bucketsPending":1,
> "copyTableCompletionPercentage":100,
> "numTablets":32,
> "numRows":80081344,
> "totalSize":4937154560
> }
> ]
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6593) unordered receivers for broadcast senders dont rerpot memmory consumption
[
https://issues.apache.org/jira/browse/DRILL-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6593:
-
Fix Version/s: (was: 1.15.0)
1.16.0
> unordered receivers for broadcast senders dont rerpot memmory consumption
> -
>
> Key: DRILL-6593
> URL: https://issues.apache.org/jira/browse/DRILL-6593
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.14.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: TPCDS_78_3_id_24c43954-65b4-07b6-7e53-a37ad47fa963.json
>
>
> In my regression test on TPCDS SF100 dataset, query 78 profile shows the
> following:
> {code}
> 05-xx-02 PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,787,488 3MB 3MB
> 05-xx-03 HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
> 0.000s 6.87% 0.32% 69,186,507 8MB 10MB
> 05-xx-04 UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
> 0.000s 0.000s 0.00% 0.00% 4,382,940 - -
> 05-xx-05 PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,803,567 3MB 3MB
> 05-xx-06 SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
> 0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
> {code}
> Note 05-xx-04 did not show memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6593) unordered receivers for broadcast senders dont rerpot memmory consumption
[
https://issues.apache.org/jira/browse/DRILL-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker reassigned DRILL-6593:
Assignee: Karthikeyan Manivannan (was: salim achouche)
> unordered receivers for broadcast senders dont rerpot memmory consumption
> -
>
> Key: DRILL-6593
> URL: https://issues.apache.org/jira/browse/DRILL-6593
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.14.0
> Environment: RHEL 7
>Reporter: Dechang Gu
>Assignee: Karthikeyan Manivannan
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: TPCDS_78_3_id_24c43954-65b4-07b6-7e53-a37ad47fa963.json
>
>
> In my regression test on TPCDS SF100 dataset, query 78 profile shows the
> following:
> {code}
> 05-xx-02 PROJECT 0.000s 0.001s 0.003s 0.022s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,787,488 3MB 3MB
> 05-xx-03 HASH_JOIN 0.000s 0.000s 0.774s 1.002s 0.000s 0.000s
> 0.000s 6.87% 0.32% 69,186,507 8MB 10MB
> 05-xx-04 UNORDERED_RECEIVER 0.000s 0.000s 0.000s 0.000s 0.000s
> 0.000s 0.000s 0.00% 0.00% 4,382,940 - -
> 05-xx-05 PROJECT 0.000s 0.001s 0.002s 0.015s 0.000s 0.000s 0.000s
> 0.02% 0.00% 64,803,567 3MB 3MB
> 05-xx-06 SELECTION_VECTOR_REMOVER0.000s 0.000s 0.333s 0.566s
> 0.000s 0.000s 0.000s 2.95% 0.14% 64,803,567 5MB 5MB
> {code}
> Note 05-xx-04 did not show memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-2035) Add ability to cancel multiple queries
[ https://issues.apache.org/jira/browse/DRILL-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-2035: - Fix Version/s: (was: 1.15.0) > Add ability to cancel multiple queries > -- > > Key: DRILL-2035 > URL: https://issues.apache.org/jira/browse/DRILL-2035 > Project: Apache Drill > Issue Type: New Feature > Components: Client - HTTP, Web Server >Reporter: Neeraja >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.16.0 > > > Currently Drill UI allows canceling one query at a time. > This could be cumbersome to manage for scenarios using with BI tools which > generate multiple queries for a single action in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6838) Query with Runtime Filter fails with IllegalStateException: Memory was leaked by query
[
https://issues.apache.org/jira/browse/DRILL-6838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682903#comment-16682903
]
weijie.tong commented on DRILL-6838:
[~aravi5] the issue has been fixed at the latest commit of DRILL-6792. The
reason is the FragmentContextImpl retained one more time reference of the
Drillbuf at the broadcast partition case.
> Query with Runtime Filter fails with IllegalStateException: Memory was leaked
> by query
> --
>
> Key: DRILL-6838
> URL: https://issues.apache.org/jira/browse/DRILL-6838
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.15.0
> Environment: Drill 1.15 + PR ->
> [https://github.com/apache/drill/pull/1504]
>Reporter: Abhishek Ravi
>Assignee: weijie.tong
>Priority: Major
> Attachments: RuntimeFilterMemoryLeak.log
>
>
> When running *TPC-H* queries with Runtime Filter feature enabled on DRILL
> 1.15.0 SNAPSHOT, I saw failures with following error:
> {noformat}
> Error: SYSTEM ERROR: IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(frag:5:38) 600/1048576/77833920/50062914560
> (res/actual/peak/limit)
> Fragment 5:38
> Please, refer to logs for more information.
> [Error Id: 2ca07049-ac92-4944-929c-16f261e43f7e on qa-node185.qa.lab:31010]
> (state=,code=0)
> {noformat}
> The failure is observed when the query has nested joins. (TPC-H query 7 can
> easily reproduce the issue). I have attached logs for a simpler sample query
> that fails.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-5956) Add storage plugin for Druid
[ https://issues.apache.org/jira/browse/DRILL-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] weijie.tong reassigned DRILL-5956: -- Assignee: (was: weijie.tong) > Add storage plugin for Druid > > > Key: DRILL-5956 > URL: https://issues.apache.org/jira/browse/DRILL-5956 > Project: Apache Drill > Issue Type: Wish > Components: Storage - Other >Reporter: Jiaqi Liu >Priority: Major > > As more and more companies are using Druid for mission-critical industrial > products, Drill could gain much more popularity with Druid as one of its > supported storage plugin so that uses could easily bind Druid cluster to > running Drill instance -- This message was sent by Atlassian JIRA (v7.6.3#76005)
