[jira] [Created] (DRILL-6902) Extra limit operator is not needed
Robert Hou created DRILL-6902:
-
Summary: Extra limit operator is not needed
Key: DRILL-6902
URL: https://issues.apache.org/jira/browse/DRILL-6902
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.15.0
Reporter: Robert Hou
Assignee: Pritesh Maker
For TPCDS query 49, there is an extra limit operator that is not needed.
Here is the query:
{noformat}
SELECT 'web' AS channel,
web.item,
web.return_ratio,
web.return_rank,
web.currency_rank
FROM (SELECT item,
return_ratio,
currency_ratio,
Rank()
OVER (
ORDER BY return_ratio) AS return_rank,
Rank()
OVER (
ORDER BY currency_ratio) AS currency_rank
FROM (SELECT ws.ws_item_sk AS
item,
( Cast(Sum(COALESCE(wr.wr_return_quantity, 0)) AS
DEC(15,
4)) /
Cast(
Sum(COALESCE(ws.ws_quantity, 0)) AS DEC(15, 4)) ) AS
return_ratio,
( Cast(Sum(COALESCE(wr.wr_return_amt, 0)) AS DEC(15, 4))
/ Cast(
Sum(
COALESCE(ws.ws_net_paid, 0)) AS DEC(15,
4)) ) AS
currency_ratio
FROM web_sales ws
LEFT OUTER JOIN web_returns wr
ON ( ws.ws_order_number =
wr.wr_order_number
AND ws.ws_item_sk = wr.wr_item_sk ),
date_dim
WHERE wr.wr_return_amt > 1
AND ws.ws_net_profit > 1
AND ws.ws_net_paid > 0
AND ws.ws_quantity > 0
AND ws_sold_date_sk = d_date_sk
AND d_year = 1999
AND d_moy = 12
GROUP BY ws.ws_item_sk) in_web) web
WHERE ( web.return_rank <= 10
OR web.currency_rank <= 10 )
UNION
SELECT 'catalog' AS channel,
catalog.item,
catalog.return_ratio,
catalog.return_rank,
catalog.currency_rank
FROM (SELECT item,
return_ratio,
currency_ratio,
Rank()
OVER (
ORDER BY return_ratio) AS return_rank,
Rank()
OVER (
ORDER BY currency_ratio) AS currency_rank
FROM (SELECT cs.cs_item_sk AS
item,
( Cast(Sum(COALESCE(cr.cr_return_quantity, 0)) AS
DEC(15,
4)) /
Cast(
Sum(COALESCE(cs.cs_quantity, 0)) AS DEC(15, 4)) ) AS
return_ratio,
( Cast(Sum(COALESCE(cr.cr_return_amount, 0)) AS DEC(15,
4
)) /
Cast(Sum(
COALESCE(cs.cs_net_paid, 0)) AS DEC(
15, 4)) ) AS
currency_ratio
FROM catalog_sales cs
LEFT OUTER JOIN catalog_returns cr
ON ( cs.cs_order_number =
cr.cr_order_number
AND cs.cs_item_sk = cr.cr_item_sk ),
date_dim
WHERE cr.cr_return_amount > 1
AND cs.cs_net_profit > 1
AND cs.cs_net_paid > 0
AND cs.cs_quantity > 0
AND cs_sold_date_sk = d_date_sk
AND d_year = 1999
AND d_moy = 12
GROUP BY cs.cs_item_sk) in_cat) catalog
WHERE ( catalog.return_rank <= 10
OR catalog.currency_rank <= 10 )
UNION
SELECT 'store' AS channel,
store.item,
store.return_ratio,
store.return_rank,
store.currency_rank
FROM (SELECT item,
return_ratio,
currency_ratio,
Rank()
OVER (
ORDER BY return_ratio) AS return_rank,
Rank()
OVER (
ORDER BY currency_ratio) AS currency_rank
FROM (SELECT sts.ss_item_sk AS
item,
( Cast(Sum(COALESCE(sr.sr_return_quantity, 0)
[jira] [Created] (DRILL-6901) Move SchemaBuilder from test to main for use outside tests
Paul Rogers created DRILL-6901:
--
Summary: Move SchemaBuilder from test to main for use outside tests
Key: DRILL-6901
URL: https://issues.apache.org/jira/browse/DRILL-6901
Project: Apache Drill
Issue Type: Improvement
Affects Versions: 1.14.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Fix For: 1.15.0
The {{SchemaBuilder}} class provides a fluent way to build a {{TupleMetadata}}.
It currently resides in the test name space as it was original used only for
tests. Move it into a "main" package for use by regular code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6900) maven-release-plugin failure on the release:perform stage
Vitalii Diravka created DRILL-6900:
--
Summary: maven-release-plugin failure on the release:perform stage
Key: DRILL-6900
URL: https://issues.apache.org/jira/browse/DRILL-6900
Project: Apache Drill
Issue Type: Bug
Components: Tools, Build & Test
Affects Versions: 1.14.0
Reporter: Vitalii Diravka
Assignee: Vitalii Diravka
Fix For: 1.15.0
{{release:perform}} stage fails on current master branch. It blocs to prepare
the release candidate: {code}
{{mvn release:perform
-DconnectionUrl=scm:git:g...@github.com:vkorukanti/drill.git -DskipTests
-Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests=true
-DconnectionUrl=scm:git:g...@github.com:vkorukanti/drill.git"}}
{code}
{code}
[INFO] [ERROR]
/home/vitalii/src/release/drill/target/checkout/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/impl/hive/AbstractDrillPrimitiveObjectInspector.java:[26,47]
error: cannot find symbol
[INFO] symbol: class PrimitiveTypeInfo
[INFO] location: class AbstractDrillPrimitiveObjectInspector
[INFO] [INFO] 100 errors
[INFO] [INFO] -
[INFO] [INFO]
[INFO] [INFO] Reactor Summary for Apache Drill Root POM 1.15.0:
[INFO] [INFO]
[INFO] [INFO] Apache Drill Root POM .. SUCCESS [
37.579 s]
[INFO] [INFO] tools/Parent Pom ... SUCCESS [
6.360 s]
[INFO] [INFO] tools/freemarker codegen tooling ... SUCCESS [
26.726 s]
[INFO] [INFO] Drill Protocol . SUCCESS [
41.030 s]
[INFO] [INFO] Common (Logical Plan, Base expressions) SUCCESS [
27.754 s]
[INFO] [INFO] Logical Plan, Base expressions . SUCCESS [
29.154 s]
[INFO] [INFO] exec/Parent Pom SUCCESS [
6.252 s]
[INFO] [INFO] exec/memory/Parent Pom . SUCCESS [
6.074 s]
[INFO] [INFO] exec/memory/base ... SUCCESS [
24.909 s]
[INFO] [INFO] exec/rpc ... SUCCESS [
24.191 s]
[INFO] [INFO] exec/Vectors ... SUCCESS
[01:50 min]
[INFO] [INFO] contrib/Parent Pom . SUCCESS [
6.173 s]
[INFO] [INFO] contrib/data/Parent Pom SUCCESS [
6.058 s]
[INFO] [INFO] contrib/data/tpch-sample-data .. SUCCESS [
19.704 s]
[INFO] [INFO] exec/Java Execution Engine . SUCCESS
[01:57 min]
[INFO] [INFO] exec/JDBC Driver using dependencies SUCCESS [
32.619 s]
[INFO] [INFO] JDBC JAR with all dependencies . FAILURE
[01:16 min]
[INFO] [INFO] Drill-on-YARN .. SKIPPED
[INFO] [INFO] contrib/kudu-storage-plugin SKIPPED
[INFO] [INFO] contrib/opentsdb-storage-plugin SKIPPED
[INFO] [INFO] contrib/mongo-storage-plugin ... SKIPPED
[INFO] [INFO] contrib/hbase-storage-plugin ... SKIPPED
[INFO] [INFO] contrib/jdbc-storage-plugin SKIPPED
[INFO] [INFO] contrib/hive-storage-plugin/Parent Pom . SKIPPED
[INFO] [INFO] contrib/hive-storage-plugin/hive-exec-shaded ... SKIPPED
[INFO] [INFO] contrib/mapr-format-plugin . SKIPPED
[INFO] [INFO] contrib/hive-storage-plugin/core ... SKIPPED
[INFO] [INFO] contrib/kafka-storage-plugin ... SKIPPED
[INFO] [INFO] contrib/drill-udfs . SKIPPED
[INFO] [INFO] Packaging and Distribution Assembly SKIPPED
[INFO] [INFO]
[INFO] [INFO] BUILD FAILURE
[INFO] [INFO]
[INFO] [INFO] Total time: 09:58 min
[INFO] [INFO] Finished at: 2018-12-12T19:18:08+02:00
[INFO] [INFO]
[INFO] [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project drill-storage-hive-core: Compilation failure: Compilation failure:
[INFO] [ERROR]
/home/vitalii/src/release/drill/target/checkout/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[32,36]
error: package org.apache.hadoop.hive.common does not exist
[INFO] [ERROR]
/home/vitalii/src/release/drill/target/checkout/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[33,34]
error: package org.apache.hadoop.hive.conf does not exist
[INFO] [ERROR]
/home/vitalii/src/relea
[jira] [Created] (DRILL-6899) Fix timestamp issues in unit tests ignored with DRILL-6833
Gautam Parai created DRILL-6899:
---
Summary: Fix timestamp issues in unit tests ignored with DRILL-6833
Key: DRILL-6899
URL: https://issues.apache.org/jira/browse/DRILL-6899
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.15.0
Reporter: Gautam Parai
Assignee: Gautam Parai
Fix For: 1.16.0
{{The following tests were disabled in the PR for DRILL-6833}}
{{IndexPlanTest.testCastTimestampPlan() - Re-enable after the MapRDB format
plugin issue is fixed.}}
{{IndexPlanTest.testRowkeyJoinPushdown_13() - Re-enable the testcase after
fixing the execution issue with HashJoin used as Rowkeyjoin.}}
{{IndexPlanTest.testRowkeyJoinPushdown_12() - Remove the testcase since the
SemiJoin transformation makes the rowkeyjoinpushdown transformation invalid.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6881) Hash-Table insert and probe: Compare hash values before keys
[ https://issues.apache.org/jira/browse/DRILL-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boaz Ben-Zvi updated DRILL-6881: Fix Version/s: (was: 1.16.0) > Hash-Table insert and probe: Compare hash values before keys > > > Key: DRILL-6881 > URL: https://issues.apache.org/jira/browse/DRILL-6881 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.14.0 >Reporter: Boaz Ben-Zvi >Assignee: Boaz Ben-Zvi >Priority: Major > > When checking for existence of a key in the hash table (during _put_ or > _probe_ operations), the value of that key is compared (using generated code) > with a potential match key (same bucket). > This comparison is slightly expensive (e.g., long keys, multi column keys, > checking null conditions, NaN, etc). Instead, if the hash-values of the two > keys are compared first (at practically zero cost), then the costly > comparison can be avoided in case the hash values don't match. > This code change is trivial, and given that the relevant Hash-Table code is > *hot code*, then even minute improvements could add up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6898) Web UI cannot be used without internet connection (jquery loaded from ajax.googleapis.com)
[ https://issues.apache.org/jira/browse/DRILL-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719135#comment-16719135 ] Arina Ielchiieva commented on DRILL-6898: - Might be fixed in https://issues.apache.org/jira/browse/DRILL-6776. > Web UI cannot be used without internet connection (jquery loaded from > ajax.googleapis.com) > -- > > Key: DRILL-6898 > URL: https://issues.apache.org/jira/browse/DRILL-6898 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.14.0 >Reporter: Paul Bormans >Priority: Major > > When opening the web ui in an environment that does not have an internet > connection, then the jquery js library is not loaded and the website does not > function as it should. > One solution can be to add a configuration option to use local/packages > javascript libraries iso loading these from a CDN. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6898) Web UI cannot be used without internet connection (jquery loaded from ajax.googleapis.com)
Paul Bormans created DRILL-6898: --- Summary: Web UI cannot be used without internet connection (jquery loaded from ajax.googleapis.com) Key: DRILL-6898 URL: https://issues.apache.org/jira/browse/DRILL-6898 Project: Apache Drill Issue Type: Improvement Components: Web Server Affects Versions: 1.14.0 Reporter: Paul Bormans When opening the web ui in an environment that does not have an internet connection, then the jquery js library is not loaded and the website does not function as it should. One solution can be to add a configuration option to use local/packages javascript libraries iso loading these from a CDN. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6893) Invalid output for star and self-join queries for RDBMS Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6893:
---
Description:
Invalid output for star and self-join queries for RDBMS Storage Plugin:
{code:java}
0: jdbc:drill:zk=local> SELECT * FROM (SELECT * FROM
mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t` INNER JOIN
(SELECT * FROM mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t0`
ON `t`.`PersonID` = `t0`.`PersonID` ;
+---+--+-+--++---+--+---+
| PersonID | OrderID | ItemID | GroupID | PersonID0 | OrderID0 | ItemID0
| GroupID0 |
+---+--+-+--++---+--+---+
| 10| 10 | 10 | 10 | null | null | null
| null |
+---+--+-+--++---+--+---+
1 row selected (1.402 seconds)
0: jdbc:drill:zk=local> select * from sys.version;
+--+---+++++
| version | commit_id |
commit_message |commit_time |
build_email | build_time |
+--+---+++++
| 1.15.0-SNAPSHOT | 100a68b314230d4cf327477f7a10f9c650720513 | DRILL-540:
Allow querying hive views in Drill | 30.11.2018 @ 10:50:46 EET |
vitalii.dira...@gmail.com | 10.12.2018 @ 15:46:54 EET |
+--+---+++++
1 row selected (0.302 seconds)
{code}
The same result in older 1.11.0 Drill version:
{code:java}
0: jdbc:drill:zk=local> SELECT * FROM (SELECT * FROM
mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t` INNER JOIN
(SELECT * FROM mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t0`
ON `t`.`PersonID` = `t0`.`PersonID`;
+---+--+-+--++---+--+---+
| PersonID | OrderID | ItemID | GroupID | PersonID0 | OrderID0 | ItemID0
| GroupID0 |
+---+--+-+--++---+--+---+
| 10| 10 | 10 | 10 | null | null | null
| null |
+---+--+-+--++---+--+---+
1 row selected (1.344 seconds)
0: jdbc:drill:zk=local> select * from sys.version;
+--+---+--+-+++
| version | commit_id |
commit_message| commit_time |
build_email | build_time |
+--+---+--+-+++
| 1.11.0 | 4220fb2fffbc81883df3e5fea575fa0a584852b3 | [maven-release-plugin]
prepare release drill-1.11.0 | 24.07.2017 @ 16:47:07 EEST |
vitalii.dira...@gmail.com | 06.12.2018 @ 14:36:39 EET |
+--+---+--+-+++
1 row selected (0.271 seconds)
{code}
But the same query in MySQL:
{code:java}
mysql> select * from `mscIdentities3` t1 join `mscIdentities3` t2 on
t1.`PersonId` = t2.`PersonID` where t1.`PersonID` = 10;
+--+-++-+--+-++-+
| PersonID | OrderID | ItemID | GroupID | PersonID | OrderID | ItemID | GroupID
|
+--+-++-+--+-++-+
| 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10
|
+--+-++-+--+-++-+
1 row in set (1.27 sec)
{code}
Drill physical plan for above query is:
{code:java}
00-00Screen : rowType = RecordType(INTEGER PersonID, INTEGER OrderID,
INTEGER ItemID, INTEGER GroupID, INTEGER PersonID0, INTEGER OrderID0, INTEGER
ItemID0, INTEGER GroupID0): rowcount = 15.0, cumulative cost = {31.5 rows,
136.5 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 735
00-01
[jira] [Updated] (DRILL-6893) Invalid output for star and self-join queries for RDBMS Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6893:
---
Summary: Invalid output for star and self-join queries for RDBMS Storage
Plugin (was: Invalid output for star and self-join queries for RDBS Storage
Plugin)
> Invalid output for star and self-join queries for RDBMS Storage Plugin
> --
>
> Key: DRILL-6893
> URL: https://issues.apache.org/jira/browse/DRILL-6893
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JDBC
>Affects Versions: 1.14.0
> Environment: mysql-5.7.23-0ubuntu0.18.04.1
> mysql-connector-java-5.1.39-bin.jar
>Reporter: Vitalii Diravka
>Priority: Major
> Fix For: Future
>
>
> Invalid output for star and self-join queries for RDBS Storage Plugin:
> {code}
> 0: jdbc:drill:zk=local> SELECT * FROM (SELECT * FROM
> mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t` INNER JOIN
> (SELECT * FROM mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t0`
> ON `t`.`PersonID` = `t0`.`PersonID` ;
> +---+--+-+--++---+--+---+
> | PersonID | OrderID | ItemID | GroupID | PersonID0 | OrderID0 |
> ItemID0 | GroupID0 |
> +---+--+-+--++---+--+---+
> | 10| 10 | 10 | 10 | null | null | null
> | null |
> +---+--+-+--++---+--+---+
> 1 row selected (1.402 seconds)
> 0: jdbc:drill:zk=local> select * from sys.version;
> +--+---+++++
> | version | commit_id |
>commit_message |commit_time |
> build_email | build_time |
> +--+---+++++
> | 1.15.0-SNAPSHOT | 100a68b314230d4cf327477f7a10f9c650720513 | DRILL-540:
> Allow querying hive views in Drill | 30.11.2018 @ 10:50:46 EET |
> vitalii.dira...@gmail.com | 10.12.2018 @ 15:46:54 EET |
> +--+---+++++
> 1 row selected (0.302 seconds)
> {code}
> The same result in older 1.11.0 Drill version:
> {code}
> 0: jdbc:drill:zk=local> SELECT * FROM (SELECT * FROM
> mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t` INNER JOIN
> (SELECT * FROM mysql.`testdb`.`mscIdentities3` WHERE `PersonID` = 10) AS `t0`
> ON `t`.`PersonID` = `t0`.`PersonID`;
> +---+--+-+--++---+--+---+
> | PersonID | OrderID | ItemID | GroupID | PersonID0 | OrderID0 |
> ItemID0 | GroupID0 |
> +---+--+-+--++---+--+---+
> | 10| 10 | 10 | 10 | null | null | null
> | null |
> +---+--+-+--++---+--+---+
> 1 row selected (1.344 seconds)
> 0: jdbc:drill:zk=local> select * from sys.version;
> +--+---+--+-+++
> | version | commit_id |
> commit_message| commit_time |
> build_email | build_time |
> +--+---+--+-+++
> | 1.11.0 | 4220fb2fffbc81883df3e5fea575fa0a584852b3 |
> [maven-release-plugin] prepare release drill-1.11.0 | 24.07.2017 @ 16:47:07
> EEST | vitalii.dira...@gmail.com | 06.12.2018 @ 14:36:39 EET |
> +--+---+--+-+++
> 1 row selected (0.271 seconds)
> {code}
> But the same query in MySQL:
> {code}
> mysql> select * from `mscIdentities3` t1 join `mscIdentities3` t2 on
> t1.`PersonId` = t2.
[jira] [Commented] (DRILL-6878) Index Plan Test failures after DRILL-6833
[
https://issues.apache.org/jira/browse/DRILL-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718911#comment-16718911
]
Vitalii Diravka commented on DRILL-6878:
Merged to Apache Drill master branch with commit id b954c57c
> Index Plan Test failures after DRILL-6833
> -
>
> Key: DRILL-6878
> URL: https://issues.apache.org/jira/browse/DRILL-6878
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Gautam Parai
>Assignee: Gautam Parai
>Priority: Blocker
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> [ERROR] Failures:
> [ERROR] IndexPlanTest.testCastTimestampPlan:1148 Incorrect number of rows
> returned by query. expected:<1> but was:<0>
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_1:1702 Did not find expected
> pattern in plan: RowKeyJoin
> 00-00 Screen
> 00-01 Project(ssn=[$0])
> 00-02 Project(ssn=[$1])
> 00-03 HashJoin(condition=[=($0, $2)], joinType=[inner], semi-join: =[true])
> 00-05 Project(_id=[$0], ITEM=[ITEM($1, 'ssn')])
> 00-06 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`_id`, `id`.`ssn`], maxwidth=2]])
> 00-04 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=($0 =
> {"$date":"2013-02-04T22:34:38.000Z"}
> ), indexName=hash_i_cast_timestamp_firstlogin], columns=[`_id`],
> maxwidth=10]])
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_10:1883 Incorrect number of rows
> returned by query. expected:<1> but was:<0>
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_12:1918 Did not find expected
> pattern in plan: HashJoin(.*[
> ])+.*Scan.*indexName=i_state_city_dl(.*[
> ])+.*RowKeyJoin(.*[
> ])+.*RestrictedJsonTableGroupScan(.*[
> ])+.*HashAgg(group=[\{0}])(.*[
> ])+.*HashJoin
> 00-00 Screen
> 00-01 Project(ssn=[$0])
> 00-02 Project(ssn=[$1])
> 00-03 HashJoin(condition=[=($0, $5)], joinType=[inner], semi-join: =[true])
> 00-05 HashJoin(condition=[=($2, $3)], joinType=[inner], semi-join: =[false])
> 00-08 Project(_id=[$0], ITEM=[ITEM($1, 'ssn')], ITEM2=[ITEM($2, 'city')])
> 00-11 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`_id`, `id`.`ssn`, `address`.`city`], maxwidth=2]])
> 00-07 Project(ITEM0=[$0], ITEM1=[$1])
> 00-10 Project(ITEM=[ITEM($0, 'city')], ITEM1=[ITEM($0, 'state')])
> 00-14 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=(address.state = "pc"), indexName=i_state_city_dl],
> columns=[`address`.`city`, `address`.`state`], maxwidth=1]])
> 00-04 Project(EXPR$0=[CAST(CAST($0):INTEGER):VARCHAR(10) CHARACTER SET
> "UTF-16LE" COLLATE "UTF-16LE$en_US$primary"])
> 00-06 Project(rowid=[$1], ITEM=[$2], ITEM2=[$3], ITEM0=[$0])
> 00-09 HashJoin(condition=[=($2, $0)], joinType=[inner], semi-join: =[false])
> 00-13 Project(ITEM=[ITEM($0, 'city')])
> 00-16 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`address`.`city`], maxwidth=2]])
> 00-12 Project(rowid=[$0], ITEM0=[$1], ITEM2=[$2])
> 00-15 SelectionVectorRemover
> 00-17 Filter(condition=[=(CAST($2):TIMESTAMP(0), 2013-02-04 22:34:38)])
> 00-18 Project(rowid=[$0], ITEM=[ITEM($1, 'city')], ITEM2=[ITEM(ITEM($2,
> 'irs'), 'firstlogin')])
> 00-19 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`rowid`, `address`.`city`,
> `activity`.`irs`.`firstlogin`], maxwidth=2]])
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_6:1799 Did not find expected
> pattern in plan: RowKeyJoin
> 00-00 Screen
> 00-01 Project(ssn=[$0])
> 00-02 Project(ssn=[$1])
> 00-03 HashJoin(condition=[=($0, $2)], joinType=[inner], semi-join: =[true])
> 00-05 Project(_id=[$0], ITEM=[ITEM($1, 'ssn')])
> 00-07 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`_id`, `id`.`ssn`], maxwidth=2]])
> 00-04 Project(EXPR$0=[CAST(CAST($0):INTEGER):VARCHAR(10) CHARACTER SET
> "UTF-16LE" COLLATE "UTF-16LE$en_US$primary"])
> 00-06 Project(rowid=[$1], ITEM=[$2], ITEM2=[$3], ITEM0=[$0])
> 00-08 HashJoin(condition=[=($2, $0)], joinType=[inner], semi-join: =[false])
> 00-10 Project(ITEM=[ITEM($0, 'city')])
> 00-12 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=Js
[jira] [Updated] (DRILL-6878) Index Plan Test failures after DRILL-6833
[
https://issues.apache.org/jira/browse/DRILL-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka updated DRILL-6878:
---
Labels: ready-to-commit (was: )
> Index Plan Test failures after DRILL-6833
> -
>
> Key: DRILL-6878
> URL: https://issues.apache.org/jira/browse/DRILL-6878
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Gautam Parai
>Assignee: Gautam Parai
>Priority: Blocker
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> [ERROR] Failures:
> [ERROR] IndexPlanTest.testCastTimestampPlan:1148 Incorrect number of rows
> returned by query. expected:<1> but was:<0>
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_1:1702 Did not find expected
> pattern in plan: RowKeyJoin
> 00-00 Screen
> 00-01 Project(ssn=[$0])
> 00-02 Project(ssn=[$1])
> 00-03 HashJoin(condition=[=($0, $2)], joinType=[inner], semi-join: =[true])
> 00-05 Project(_id=[$0], ITEM=[ITEM($1, 'ssn')])
> 00-06 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`_id`, `id`.`ssn`], maxwidth=2]])
> 00-04 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=($0 =
> {"$date":"2013-02-04T22:34:38.000Z"}
> ), indexName=hash_i_cast_timestamp_firstlogin], columns=[`_id`],
> maxwidth=10]])
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_10:1883 Incorrect number of rows
> returned by query. expected:<1> but was:<0>
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_12:1918 Did not find expected
> pattern in plan: HashJoin(.*[
> ])+.*Scan.*indexName=i_state_city_dl(.*[
> ])+.*RowKeyJoin(.*[
> ])+.*RestrictedJsonTableGroupScan(.*[
> ])+.*HashAgg(group=[\{0}])(.*[
> ])+.*HashJoin
> 00-00 Screen
> 00-01 Project(ssn=[$0])
> 00-02 Project(ssn=[$1])
> 00-03 HashJoin(condition=[=($0, $5)], joinType=[inner], semi-join: =[true])
> 00-05 HashJoin(condition=[=($2, $3)], joinType=[inner], semi-join: =[false])
> 00-08 Project(_id=[$0], ITEM=[ITEM($1, 'ssn')], ITEM2=[ITEM($2, 'city')])
> 00-11 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`_id`, `id`.`ssn`, `address`.`city`], maxwidth=2]])
> 00-07 Project(ITEM0=[$0], ITEM1=[$1])
> 00-10 Project(ITEM=[ITEM($0, 'city')], ITEM1=[ITEM($0, 'state')])
> 00-14 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=(address.state = "pc"), indexName=i_state_city_dl],
> columns=[`address`.`city`, `address`.`state`], maxwidth=1]])
> 00-04 Project(EXPR$0=[CAST(CAST($0):INTEGER):VARCHAR(10) CHARACTER SET
> "UTF-16LE" COLLATE "UTF-16LE$en_US$primary"])
> 00-06 Project(rowid=[$1], ITEM=[$2], ITEM2=[$3], ITEM0=[$0])
> 00-09 HashJoin(condition=[=($2, $0)], joinType=[inner], semi-join: =[false])
> 00-13 Project(ITEM=[ITEM($0, 'city')])
> 00-16 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`address`.`city`], maxwidth=2]])
> 00-12 Project(rowid=[$0], ITEM0=[$1], ITEM2=[$2])
> 00-15 SelectionVectorRemover
> 00-17 Filter(condition=[=(CAST($2):TIMESTAMP(0), 2013-02-04 22:34:38)])
> 00-18 Project(rowid=[$0], ITEM=[ITEM($1, 'city')], ITEM2=[ITEM(ITEM($2,
> 'irs'), 'firstlogin')])
> 00-19 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`rowid`, `address`.`city`,
> `activity`.`irs`.`firstlogin`], maxwidth=2]])
> [ERROR] IndexPlanTest.testRowkeyJoinPushdown_6:1799 Did not find expected
> pattern in plan: RowKeyJoin
> 00-00 Screen
> 00-01 Project(ssn=[$0])
> 00-02 Project(ssn=[$1])
> 00-03 HashJoin(condition=[=($0, $2)], joinType=[inner], semi-join: =[true])
> 00-05 Project(_id=[$0], ITEM=[ITEM($1, 'ssn')])
> 00-07 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], columns=[`_id`, `id`.`ssn`], maxwidth=2]])
> 00-04 Project(EXPR$0=[CAST(CAST($0):INTEGER):VARCHAR(10) CHARACTER SET
> "UTF-16LE" COLLATE "UTF-16LE$en_US$primary"])
> 00-06 Project(rowid=[$1], ITEM=[$2], ITEM2=[$3], ITEM0=[$0])
> 00-08 HashJoin(condition=[=($2, $0)], joinType=[inner], semi-join: =[false])
> 00-10 Project(ITEM=[ITEM($0, 'city')])
> 00-12 Scan(table=[[hbase, index_test_primary]], groupscan=[JsonTableGroupScan
> [ScanSpec=JsonScanSpec [tableName=maprfs:///tmp/index_test_primary,
> condition=null], column
[jira] [Commented] (DRILL-6879) Indicate a warning in the WebUI when a query makes little to no progress for a while
[
https://issues.apache.org/jira/browse/DRILL-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718815#comment-16718815
]
ASF GitHub Bot commented on DRILL-6879:
---
arina-ielchiieva commented on a change in pull request #1572: DRILL-6879: Show
warnings for potential performance issues
URL: https://github.com/apache/drill/pull/1572#discussion_r240975519
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
##
@@ -73,6 +73,10 @@ public ProfileWrapper(final QueryProfile profile,
DrillConfig drillConfig) {
final List majors = new
ArrayList<>(profile.getFragmentProfileList());
Collections.sort(majors, Comparators.majorId);
+//Setting warning thresholds for performance-degrading queries (DRILL-6879)
Review comment:
Do not use static methods, just pass drillConfig in the constructor for each
of there classes and set warning threshold value during instance creation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Indicate a warning in the WebUI when a query makes little to no progress for
> a while
>
>
> Key: DRILL-6879
> URL: https://issues.apache.org/jira/browse/DRILL-6879
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Monitoring, Web Server
>Affects Versions: 1.14.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: user-experience
> Fix For: 1.16.0
>
> Attachments: image-2018-12-04-11-54-54-247.png,
> image-2018-12-06-11-19-00-339.png, image-2018-12-06-11-27-14-719.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> When running a very large query on a cluster with limited resource, we
> noticed that one of the node's VM thread freezes the fragment threads as it
> tries to do some work (GC perhaps?). This is a clear indication that the
> query is stuck in a weird state where it might not recover from.
> Under such circumstances, it makes sense to cancel or atleast warn the user
> on that page of the query exceeding a certain threshold.
> For detecting this, the user will find that the {{Last Progress}} column in
> the Fragments Overview section will show large times.
> !image-2018-12-04-11-54-54-247.png|width=969,height=336!
> In addition, there are instances where a query might have buffered operators
> spilling to disk, which also hits performance (and, subsequently, longer run
> times). Calling out this skew can be very useful.
> !image-2018-12-06-11-27-14-719.png|width=969,height=256!
> Or there might be cases where a single fragment takes much longer than the
> average (indicated by an extreme skew in the Gantt chart).
> !image-2018-12-06-11-19-00-339.png|width=969,height=150!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
