[jira] [Commented] (DRILL-3280) Missing OVER clause in window function query results in AssertionError
[
https://issues.apache.org/jira/browse/DRILL-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741179#comment-14741179
]
Victoria Markman commented on DRILL-3280:
-
Sean,
Case below should throw an error about rank() function requiring over clause:
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > row_number(),
. . . . . . . . . . . . > rank() over (order by a1),
. . . . . . . . . . . . > dense_rank() over(order by a1)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > ;
Error: VALIDATION ERROR: From line 3, column 31 to line 3, column 32:
Expression 'a1' is not being grouped
{code}
If I remove row_number(), query works (it does not need 'group by clause'):
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > --row_number(),
. . . . . . . . . . . . > rank() over (order by a1),
. . . . . . . . . . . . > dense_rank() over(order by a1)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > ;
+-+-+
| EXPR$0 | EXPR$1 |
+-+-+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
| 5 | 5 |
| 6 | 6 |
| 7 | 7 |
| 8 | 8 |
| 9 | 9 |
| 10 | 10 |
+-+-+
10 rows selected (0.326 seconds)
{code}
When you I run it by itself, I get a correct error message:
{code}
0: jdbc:drill:schema=dfs> select row_number() from t1;
Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 19: OVER
clause is necessary for window functions
[Error Id: 8324aa8b-758b-4ee3-b6ad-494d4a8e654d on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
> Missing OVER clause in window function query results in AssertionError
> --
>
> Key: DRILL-3280
> URL: https://issues.apache.org/jira/browse/DRILL-3280
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.0.0
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Missing OVER clause results in AssertionError.
> Instead, we will need an error message that said, "window function call
> requires an OVER clause"
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rank(), cume_dist() over w from
> `allDataInPrq/0_0_0.parquet` window w as (partition by col_chr order by
> col_dbl);
> Error: SYSTEM ERROR: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected exception during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> 2015-06-11 20:50:42,054 [2a860b5d-dd87-087f-3730-bf47a10f5d97:foreman] ERROR
> o.a.d.c.exceptions.UserException - SYSTEM ERROR:
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
> ~[drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:840)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:782)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:784)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:893)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
[jira] [Comment Edited] (DRILL-3280) Missing OVER clause in window function query results in AssertionError
[
https://issues.apache.org/jira/browse/DRILL-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741179#comment-14741179
]
Victoria Markman edited comment on DRILL-3280 at 9/11/15 5:24 PM:
--
Sean,
Case below should throw an error about rank() function requiring over clause:
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > row_number(),
. . . . . . . . . . . . > rank() over (order by a1),
. . . . . . . . . . . . > dense_rank() over(order by a1)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > ;
Error: VALIDATION ERROR: From line 3, column 31 to line 3, column 32:
Expression 'a1' is not being grouped
{code}
If I remove row_number(), query works (it does not need 'group by clause'):
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > --row_number(),
. . . . . . . . . . . . > rank() over (order by a1),
. . . . . . . . . . . . > dense_rank() over(order by a1)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > ;
+-+-+
| EXPR$0 | EXPR$1 |
+-+-+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
| 5 | 5 |
| 6 | 6 |
| 7 | 7 |
| 8 | 8 |
| 9 | 9 |
| 10 | 10 |
+-+-+
10 rows selected (0.326 seconds)
{code}
When I run it by itself, I get a correct error message:
{code}
0: jdbc:drill:schema=dfs> select row_number() from t1;
Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 19: OVER
clause is necessary for window functions
[Error Id: 8324aa8b-758b-4ee3-b6ad-494d4a8e654d on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
was (Author: vicky):
Sean,
Case below should throw an error about rank() function requiring over clause:
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > row_number(),
. . . . . . . . . . . . > rank() over (order by a1),
. . . . . . . . . . . . > dense_rank() over(order by a1)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > ;
Error: VALIDATION ERROR: From line 3, column 31 to line 3, column 32:
Expression 'a1' is not being grouped
{code}
If I remove row_number(), query works (it does not need 'group by clause'):
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > --row_number(),
. . . . . . . . . . . . > rank() over (order by a1),
. . . . . . . . . . . . > dense_rank() over(order by a1)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > ;
+-+-+
| EXPR$0 | EXPR$1 |
+-+-+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
| 5 | 5 |
| 6 | 6 |
| 7 | 7 |
| 8 | 8 |
| 9 | 9 |
| 10 | 10 |
+-+-+
10 rows selected (0.326 seconds)
{code}
When you I run it by itself, I get a correct error message:
{code}
0: jdbc:drill:schema=dfs> select row_number() from t1;
Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 19: OVER
clause is necessary for window functions
[Error Id: 8324aa8b-758b-4ee3-b6ad-494d4a8e654d on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
> Missing OVER clause in window function query results in AssertionError
> --
>
> Key: DRILL-3280
> URL: https://issues.apache.org/jira/browse/DRILL-3280
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.0.0
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Missing OVER clause results in AssertionError.
> Instead, we will need an error message that said, "window function call
> requires an OVER clause"
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rank(), cume_dist() over w from
> `allDataInPrq/0_0_0.parquet` window w as (partition by col_chr order by
> col_dbl);
> Error: SYSTEM ERROR: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected exception during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> 2015-06-11 20:50:42,054 [2a860b5d-dd87-087f-3730-bf47a10f5d97:foreman] ERROR
> o.a.d.c.exceptions.UserException - SYSTEM ERROR:
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
>
[jira] [Commented] (DRILL-3280) Missing OVER clause in window function query results in AssertionError
[
https://issues.apache.org/jira/browse/DRILL-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741339#comment-14741339
]
Victoria Markman commented on DRILL-3280:
-
Tests added in: Functional/Passing/window_functions/negative
> Missing OVER clause in window function query results in AssertionError
> --
>
> Key: DRILL-3280
> URL: https://issues.apache.org/jira/browse/DRILL-3280
> Project: Apache Drill
> Issue Type: Bug
> Components: SQL Parser
>Affects Versions: 1.0.0
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Missing OVER clause results in AssertionError.
> Instead, we will need an error message that said, "window function call
> requires an OVER clause"
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rank(), cume_dist() over w from
> `allDataInPrq/0_0_0.parquet` window w as (partition by col_chr order by
> col_dbl);
> Error: SYSTEM ERROR: org.apache.drill.exec.work.foreman.ForemanException:
> Unexpected exception during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> 2015-06-11 20:50:42,054 [2a860b5d-dd87-087f-3730-bf47a10f5d97:foreman] ERROR
> o.a.d.c.exceptions.UserException - SYSTEM ERROR:
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: null
> [Error Id: f8675256-eea9-4ca6-859c-4c0b714f27a0 on centos-02.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
> ~[drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:840)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:782)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:784)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:893)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: null
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: null
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.getRootField(SqlToRelConverter.java:3810)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.adjustInputRef(SqlToRelConverter.java:3139)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3114)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1400(SqlToRelConverter.java:180)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4061)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3489)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:274)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3944)
> ~[calcite-c
[jira] [Commented] (DRILL-3360) Window function defined within another window function
[
https://issues.apache.org/jira/browse/DRILL-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741348#comment-14741348
]
Victoria Markman commented on DRILL-3360:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
{code}
0: jdbc:drill:schema=dfs> select rank() over(order by row_number() over(order
by c1)) from t1;
Error: VALIDATION ERROR: From line 1, column 19 to line 1, column 59: ORDER BY
expression should not contain OVER clause
[Error Id: d0b6ef18-e66f-44ec-961b-f62eb1980730 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
Test added in: Functional/Passing/window_functions/negative/drill-3360*
> Window function defined within another window function
> ---
>
> Key: DRILL-3360
> URL: https://issues.apache.org/jira/browse/DRILL-3360
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
> Environment: CentOS 4 node cluster
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> Window function defined within another window function, Postgres 9.3 does not
> support this, Drill supports it and we see results being returned. We should
> not support this kind of query.
> From Postgres 9.3
> {code}
> postgres=# select rank() over(order by row_number() over(order by col_int))
> from vwOnParq_wCst;
> ERROR: window functions are not allowed in window definitions
> LINE 1: select rank() over(order by row_number() over(order by col_i...
> {code}
> From execution on Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rank() over(order by row_number()
> over(order by col_int)) from vwOnParq_wCst;
> +-+
> | EXPR$0 |
> +-+
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 8 |
> | 9 |
> | 10 |
> | 11 |
> | 12 |
> | 13 |
> | 14 |
> | 15 |
> | 16 |
> | 17 |
> | 18 |
> | 19 |
> | 20 |
> | 21 |
> | 22 |
> | 23 |
> | 24 |
> | 25 |
> | 26 |
> | 27 |
> | 28 |
> | 29 |
> | 30 |
> +-+
> 30 rows selected (0.377 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3360) Window function defined within another window function
[
https://issues.apache.org/jira/browse/DRILL-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3360.
---
> Window function defined within another window function
> ---
>
> Key: DRILL-3360
> URL: https://issues.apache.org/jira/browse/DRILL-3360
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
> Environment: CentOS 4 node cluster
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> Window function defined within another window function, Postgres 9.3 does not
> support this, Drill supports it and we see results being returned. We should
> not support this kind of query.
> From Postgres 9.3
> {code}
> postgres=# select rank() over(order by row_number() over(order by col_int))
> from vwOnParq_wCst;
> ERROR: window functions are not allowed in window definitions
> LINE 1: select rank() over(order by row_number() over(order by col_i...
> {code}
> From execution on Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rank() over(order by row_number()
> over(order by col_int)) from vwOnParq_wCst;
> +-+
> | EXPR$0 |
> +-+
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 8 |
> | 9 |
> | 10 |
> | 11 |
> | 12 |
> | 13 |
> | 14 |
> | 15 |
> | 16 |
> | 17 |
> | 18 |
> | 19 |
> | 20 |
> | 21 |
> | 22 |
> | 23 |
> | 24 |
> | 25 |
> | 26 |
> | 27 |
> | 28 |
> | 29 |
> | 30 |
> +-+
> 30 rows selected (0.377 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3601) LEAD function used without OVER clause should not plan
[
https://issues.apache.org/jira/browse/DRILL-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741361#comment-14741361
]
Victoria Markman commented on DRILL-3601:
-
Verified fixed in: 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Test added in: Functional/Passing/window_functions/negative/no_over_clause-3.sql
> LEAD function used without OVER clause should not plan
> --
>
> Key: DRILL-3601
> URL: https://issues.apache.org/jira/browse/DRILL-3601
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
> Environment: private-branch-with-new-window-functions
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
> Attachments: drillbit.log, fewRowsAllData.parquet
>
>
> When using LEAD window function without OVER clause, query should fail. Yet
> it generates a StreamingAgg operator and then fails trying to compile it's
> generated code.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select lead(col0) from `fewRowsAllData.parquet`;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> CompileException: Line 95, Column 62: Unknown variable or type "index"
> Fragment 0:0
> [Error Id: 0d967bd7-36f6-4c8d-a574-9d67d7b22f7e on centos-01.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Here is the plan for such query:
> {noformat}
> 0: jdbc:drill:> explain plan for select lead(col0) from
> `fewRowsAllData.parquet`;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[LEAD($0)])
> 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=file:/Users/hakim/MapR/data/3601.parquet]],
> selectionRoot=file:/Users/hakim/MapR/data/fewRowsAllData.parquet, numFiles=1,
> columns=[`col0`]]])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3601) LEAD function used without OVER clause should not plan
[
https://issues.apache.org/jira/browse/DRILL-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3601.
---
> LEAD function used without OVER clause should not plan
> --
>
> Key: DRILL-3601
> URL: https://issues.apache.org/jira/browse/DRILL-3601
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
> Environment: private-branch-with-new-window-functions
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
> Attachments: drillbit.log, fewRowsAllData.parquet
>
>
> When using LEAD window function without OVER clause, query should fail. Yet
> it generates a StreamingAgg operator and then fails trying to compile it's
> generated code.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select lead(col0) from `fewRowsAllData.parquet`;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> CompileException: Line 95, Column 62: Unknown variable or type "index"
> Fragment 0:0
> [Error Id: 0d967bd7-36f6-4c8d-a574-9d67d7b22f7e on centos-01.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Here is the plan for such query:
> {noformat}
> 0: jdbc:drill:> explain plan for select lead(col0) from
> `fewRowsAllData.parquet`;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[LEAD($0)])
> 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=file:/Users/hakim/MapR/data/3601.parquet]],
> selectionRoot=file:/Users/hakim/MapR/data/fewRowsAllData.parquet, numFiles=1,
> columns=[`col0`]]])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3649) LEAD , LAG , NTILE , FIRST_VALUE , LAST_VALUE report RuntimeException for missing OVER clause
[
https://issues.apache.org/jira/browse/DRILL-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741430#comment-14741430
]
Victoria Markman commented on DRILL-3649:
-
Verified fixed in: 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Test added in: Functional/Passing/window_functions/negative
> LEAD , LAG , NTILE , FIRST_VALUE , LAST_VALUE report RuntimeException for
> missing OVER clause
> -
>
> Key: DRILL-3649
> URL: https://issues.apache.org/jira/browse/DRILL-3649
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Environment: private-branch
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> Missing OVER clause must be caught at query plan time, instead we see
> RuntimeException.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select NTILE(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> SchemaChangeException: Failure while materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [ntile(INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> [Error Id: 5f2f6ffa-7557-447e-a015-63cc87e3e543 on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select FIRST_VALUE(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> SchemaChangeException: Failure while materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [first_value(INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> [Error Id: cc52f460-bd85-4588-9f1c-bcf5c6e4729c on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select LAST_VALUE(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> SchemaChangeException: Failure while materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [last_value(INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> [Error Id: b02c7c59-f9b0-4dc5-89f8-2eb754fcd27b on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select LEAD(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> CompileException: Line 89, Column 60: Unknown variable or type "index"
> Fragment 0:0
> [Error Id: c8375ab9-69ed-4f37-83a4-639d8780762e on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:
[jira] [Closed] (DRILL-3649) LEAD , LAG , NTILE , FIRST_VALUE , LAST_VALUE report RuntimeException for missing OVER clause
[
https://issues.apache.org/jira/browse/DRILL-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3649.
---
> LEAD , LAG , NTILE , FIRST_VALUE , LAST_VALUE report RuntimeException for
> missing OVER clause
> -
>
> Key: DRILL-3649
> URL: https://issues.apache.org/jira/browse/DRILL-3649
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Environment: private-branch
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> Missing OVER clause must be caught at query plan time, instead we see
> RuntimeException.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select NTILE(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> SchemaChangeException: Failure while materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [ntile(INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> [Error Id: 5f2f6ffa-7557-447e-a015-63cc87e3e543 on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select FIRST_VALUE(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> SchemaChangeException: Failure while materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [first_value(INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> [Error Id: cc52f460-bd85-4588-9f1c-bcf5c6e4729c on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select LAST_VALUE(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> SchemaChangeException: Failure while materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [last_value(INT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> [Error Id: b02c7c59-f9b0-4dc5-89f8-2eb754fcd27b on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select LEAD(1) from FEWRWSPQQ_101;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> CompileException: Line 89, Column 60: Unknown variable or type "index"
> Fragment 0:0
> [Error Id: c8375ab9-69ed-4f37-83a4-639d8780762e on centos-04.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:
[jira] [Closed] (DRILL-3555) Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail
[
https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3555.
---
> Changing defaults for planner.memory.max_query_memory_per_node causes queries
> with window function to fail
> --
>
> Key: DRILL-3555
> URL: https://issues.apache.org/jira/browse/DRILL-3555
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0, 1.2.0
> Environment: 4 Nodes. Direct Memory= 48 GB each
>Reporter: Abhishek Girish
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> Changing the default value for planner.memory.max_query_memory_per_node from
> 2 GB to anything higher causes queries with window functions to fail.
> Changed system options
> {code:sql}
> > select * from sys.options where status like '%CHANGE%';
> +---+--+-+--+-+-+---++
> | name| kind | type | status |
> num_val | string_val | bool_val | float_val |
> +---+--+-+--+-+-+---++
> | planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED |
> null| null| true | null |
> | planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED |
> 8589934592 | null| null | null |
> +---+--+-+--+-+-+---++
> 2 rows selected (0.249 seconds)
> {code}
> Query
> {code:sql}
> > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM
> > store_sales ss LIMIT 20;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> DrillRuntimeException: Adding this batch causes the total size to exceed max
> allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875.
> maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING -->
> FINISHED
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED
> 2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR
> o.a.d.e.p.i.s.SortRecordBatchBuilder - Adding this batch causes the total
> size to exceed max allowed size. Current runningBytes 1073638500, Incoming
> batchBytes 127875. maxBytes 1073741824
> 2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING -->
> FAILED
> ...
> 2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED -->
> FINISHED
> 2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: DrillRuntimeException:
> Adding this batch causes the total size to exceed max allowed size. Current
> runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> DrillRuntimeException: Adding this batch causes the total size to exceed max
> allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875.
> maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.
[jira] [Commented] (DRILL-3555) Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail
[
https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741452#comment-14741452
]
Victoria Markman commented on DRILL-3555:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
On a single node, 32 GB of direct memory with:
{code}
0: jdbc:drill:schema=dfs> select * from sys.options where status like
'%CHANGED%';
+---+--+--+--+-+-+---++
| name| kind | type | status |
num_val | string_val | bool_val | float_val |
+---+--+--+--+-+-+---++
| planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED |
null| null| true | null |
| planner.memory.max_query_memory_per_node | LONG | SESSION | CHANGED |
8589934592 | null| null | null |
+---+--+--+--+-+-+---++
2 rows selected (0.497 seconds)
{code}
Correctness was not verified here, [~agirish] is adding window functions on a
scale with verification. I consider this issue fixed.
{code}
0: jdbc:drill:schema=dfs> SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY
ss.ss_store_sk) FROM store_sales ss LIMIT 20;
++
| EXPR$0 |
++
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
| 2410564162.45 |
++
20 rows selected (126.975 seconds)
{code}
> Changing defaults for planner.memory.max_query_memory_per_node causes queries
> with window function to fail
> --
>
> Key: DRILL-3555
> URL: https://issues.apache.org/jira/browse/DRILL-3555
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0, 1.2.0
> Environment: 4 Nodes. Direct Memory= 48 GB each
>Reporter: Abhishek Girish
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> Changing the default value for planner.memory.max_query_memory_per_node from
> 2 GB to anything higher causes queries with window functions to fail.
> Changed system options
> {code:sql}
> > select * from sys.options where status like '%CHANGE%';
> +---+--+-+--+-+-+---++
> | name| kind | type | status |
> num_val | string_val | bool_val | float_val |
> +---+--+-+--+-+-+---++
> | planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED |
> null| null| true | null |
> | planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED |
> 8589934592 | null| null | null |
> +---+--+-+--+-+-+---++
> 2 rows selected (0.249 seconds)
> {code}
> Query
> {code:sql}
> > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM
> > store_sales ss LIMIT 20;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> DrillRuntimeException: Adding this batch causes the total size to exceed max
> allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875.
> maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:
[jira] [Closed] (DRILL-3297) Using rank, dense_rank, percent_rank, cume_dist, row_number window functions without OVER clause results in cryptic schema change error
[
https://issues.apache.org/jira/browse/DRILL-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3297.
---
> Using rank, dense_rank, percent_rank, cume_dist, row_number window functions
> without OVER clause results in cryptic schema change error
> ---
>
> Key: DRILL-3297
> URL: https://issues.apache.org/jira/browse/DRILL-3297
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select rank() from t1;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> org.apache.drill.exec.exception.SchemaChangeException: Failure while
> materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [rank(BIGINT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3297) Using rank, dense_rank, percent_rank, cume_dist, row_number window functions without OVER clause results in cryptic schema change error
[
https://issues.apache.org/jira/browse/DRILL-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741455#comment-14741455
]
Victoria Markman commented on DRILL-3297:
-
Verified fixed in: 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Test added in: Functional/Passing/window_functions/negative
> Using rank, dense_rank, percent_rank, cume_dist, row_number window functions
> without OVER clause results in cryptic schema change error
> ---
>
> Key: DRILL-3297
> URL: https://issues.apache.org/jira/browse/DRILL-3297
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select rank() from t1;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> org.apache.drill.exec.exception.SchemaChangeException: Failure while
> materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [rank(BIGINT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2830) Self-Join via view gives wrong results
[
https://issues.apache.org/jira/browse/DRILL-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741461#comment-14741461
]
Victoria Markman commented on DRILL-2830:
-
I'm removing fix version from this bug, since this bug is marked as a duplicate
of a bug that has not been fixed yet.
So DRILL-2345 will have target fix version set.
> Self-Join via view gives wrong results
> --
>
> Key: DRILL-2830
> URL: https://issues.apache.org/jira/browse/DRILL-2830
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
>
> Create a view by:
> {code}
> create view v1(x, y) as select n_regionkey, n_nationkey from
> cp.`tpch/nation.parquet`;
> {code}
> And join this view with the file which created the view with condition
> t.n_nationkey = v1.y (where 'y' is n_nationkey)
> {code}
> select t.n_nationkey from cp.`tpch/nation.parquet` t inner join v1 on
> t.n_nationkey = v1.y;
> +-+
> | n_nationkey |
> +-+
> | 0 |
> | 1 |
> | 1 |
> | 1 |
> | 4 |
> | 0 |
> | 3 |
> | 3 |
> | 2 |
> | 2 |
> | 4 |
> | 4 |
> | 2 |
> | 4 |
> | 0 |
> | 0 |
> | 0 |
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 2 |
> | 3 |
> | 3 |
> | 1 |
> +-+
> 25 rows selected (0.153 seconds)
> {code}
> After investigating the plan, I found out that the result was produced as if
> the join condition was t.n_nationkey = v1.x (where x is 'n_regionkey')
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2830) Self-Join via view gives wrong results
[
https://issues.apache.org/jira/browse/DRILL-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2830:
Fix Version/s: (was: 1.2.0)
> Self-Join via view gives wrong results
> --
>
> Key: DRILL-2830
> URL: https://issues.apache.org/jira/browse/DRILL-2830
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
>
> Create a view by:
> {code}
> create view v1(x, y) as select n_regionkey, n_nationkey from
> cp.`tpch/nation.parquet`;
> {code}
> And join this view with the file which created the view with condition
> t.n_nationkey = v1.y (where 'y' is n_nationkey)
> {code}
> select t.n_nationkey from cp.`tpch/nation.parquet` t inner join v1 on
> t.n_nationkey = v1.y;
> +-+
> | n_nationkey |
> +-+
> | 0 |
> | 1 |
> | 1 |
> | 1 |
> | 4 |
> | 0 |
> | 3 |
> | 3 |
> | 2 |
> | 2 |
> | 4 |
> | 4 |
> | 2 |
> | 4 |
> | 0 |
> | 0 |
> | 0 |
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 2 |
> | 3 |
> | 3 |
> | 1 |
> +-+
> 25 rows selected (0.153 seconds)
> {code}
> After investigating the plan, I found out that the result was produced as if
> the join condition was t.n_nationkey = v1.x (where x is 'n_regionkey')
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3653) Assert in a query with both avg aggregate and avg window aggregate functions
[
https://issues.apache.org/jira/browse/DRILL-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741484#comment-14741484
]
Victoria Markman commented on DRILL-3653:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Tests added under: Functional/Passing/window_functions/multiple_partitions
> Assert in a query with both avg aggregate and avg window aggregate functions
>
>
> Key: DRILL-3653
> URL: https://issues.apache.org/jira/browse/DRILL-3653
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_funcion
> Fix For: 1.2.0
>
> Attachments: t1_parquet
>
>
> It seems to be the problem with just this combination and I can't believe I
> did not find it earlier ...
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), avg(a1) over () from t1 group by a1;
> Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated:
> conversion result not null
> [Error Id: 2f850005-a7f6-4215-bbc1-90da57cbb71f on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Works:
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), sum(a1) over () from t1 group by a1;
> +-+-+
> | EXPR$0 | EXPR$1 |
> +-+-+
> | 1.0 | 47 |
> | 2.0 | 47 |
> | 3.0 | 47 |
> | 4.0 | 47 |
> | 5.0 | 47 |
> | 6.0 | 47 |
> | 7.0 | 47 |
> | 9.0 | 47 |
> | 10.0| 47 |
> | null| 47 |
> +-+-+
> 10 rows selected (0.54 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), count(a1) over () from t1 group by
> a1;
> +-+-+
> | EXPR$0 | EXPR$1 |
> +-+-+
> | 1.0 | 9 |
> | 2.0 | 9 |
> | 3.0 | 9 |
> | 4.0 | 9 |
> | 5.0 | 9 |
> | 6.0 | 9 |
> | 7.0 | 9 |
> | 9.0 | 9 |
> | 10.0| 9 |
> | null| 9 |
> +-+-+
> 10 rows selected (0.304 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), count(a1) over (), sum(a1)
> over(partition by b1) from t1 group by a1, b1;
> +-+-+-+
> | EXPR$0 | EXPR$1 | EXPR$2 |
> +-+-+-+
> | 1.0 | 9 | 1 |
> | 2.0 | 9 | 2 |
> | 3.0 | 9 | 3 |
> | 5.0 | 9 | 5 |
> | 6.0 | 9 | 6 |
> | 7.0 | 9 | 7 |
> | null| 9 | null|
> | 9.0 | 9 | 9 |
> | 10.0| 9 | 10 |
> | 4.0 | 9 | 4 |
> +-+-+-+
> 10 rows selected (0.788 seconds)
> {code}
> drillbit.log
> {code}
> 2015-08-15 00:06:20,260 [2a317d83-5099-1d1f-ba86-33173f88b00c:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING -->
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: Internal error: while converting
> `CastHigh`(SUM(`t1`.`a1`)) / COUNT(`t1`.`a1`)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.AssertionError: Internal error: while converting
> `CastHigh`(SUM(`t1`.`a1`)) / COUNT(`t1`.`a1`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$1.convertCall(ReflectiveConvertletTable.java:96)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4070)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3508)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3963)
>
[jira] [Closed] (DRILL-3653) Assert in a query with both avg aggregate and avg window aggregate functions
[
https://issues.apache.org/jira/browse/DRILL-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3653.
---
> Assert in a query with both avg aggregate and avg window aggregate functions
>
>
> Key: DRILL-3653
> URL: https://issues.apache.org/jira/browse/DRILL-3653
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_funcion
> Fix For: 1.2.0
>
> Attachments: t1_parquet
>
>
> It seems to be the problem with just this combination and I can't believe I
> did not find it earlier ...
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), avg(a1) over () from t1 group by a1;
> Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated:
> conversion result not null
> [Error Id: 2f850005-a7f6-4215-bbc1-90da57cbb71f on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Works:
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), sum(a1) over () from t1 group by a1;
> +-+-+
> | EXPR$0 | EXPR$1 |
> +-+-+
> | 1.0 | 47 |
> | 2.0 | 47 |
> | 3.0 | 47 |
> | 4.0 | 47 |
> | 5.0 | 47 |
> | 6.0 | 47 |
> | 7.0 | 47 |
> | 9.0 | 47 |
> | 10.0| 47 |
> | null| 47 |
> +-+-+
> 10 rows selected (0.54 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), count(a1) over () from t1 group by
> a1;
> +-+-+
> | EXPR$0 | EXPR$1 |
> +-+-+
> | 1.0 | 9 |
> | 2.0 | 9 |
> | 3.0 | 9 |
> | 4.0 | 9 |
> | 5.0 | 9 |
> | 6.0 | 9 |
> | 7.0 | 9 |
> | 9.0 | 9 |
> | 10.0| 9 |
> | null| 9 |
> +-+-+
> 10 rows selected (0.304 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select avg(a1), count(a1) over (), sum(a1)
> over(partition by b1) from t1 group by a1, b1;
> +-+-+-+
> | EXPR$0 | EXPR$1 | EXPR$2 |
> +-+-+-+
> | 1.0 | 9 | 1 |
> | 2.0 | 9 | 2 |
> | 3.0 | 9 | 3 |
> | 5.0 | 9 | 5 |
> | 6.0 | 9 | 6 |
> | 7.0 | 9 | 7 |
> | null| 9 | null|
> | 9.0 | 9 | 9 |
> | 10.0| 9 | 10 |
> | 4.0 | 9 | 4 |
> +-+-+-+
> 10 rows selected (0.788 seconds)
> {code}
> drillbit.log
> {code}
> 2015-08-15 00:06:20,260 [2a317d83-5099-1d1f-ba86-33173f88b00c:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING -->
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: Internal error: while converting
> `CastHigh`(SUM(`t1`.`a1`)) / COUNT(`t1`.`a1`)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.AssertionError: Internal error: while converting
> `CastHigh`(SUM(`t1`.`a1`)) / COUNT(`t1`.`a1`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$1.convertCall(ReflectiveConvertletTable.java:96)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4070)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3508)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3963)
> ~[calcite-core-1.1.0-drill-r16.jar:1.1.0-drill-r16]
> at
> org.apache.drill.exec.planner.sql.DrillAvgVarianceConvertlet.convertCall(DrillAvgVarianceConvertlet.java:72)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.
[jira] [Updated] (DRILL-3769) to_date function with one argument returns wrong data type
[
https://issues.apache.org/jira/browse/DRILL-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3769:
Affects Version/s: 1.2.0
1.1.0
> to_date function with one argument returns wrong data type
> --
>
> Key: DRILL-3769
> URL: https://issues.apache.org/jira/browse/DRILL-3769
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Victoria Markman
>
> 1. to_date function is not part of SQL standard according to my research
> (checked ISO/IEC9075-2), so implementations of it may vary from database to
> database (our implementation of to_date is different from Postgres, for
> example)
> 2. Our documentation only talks about to_date with 2 parameters: format and
> actual string to be converted to date type
> 3. Calcite does not seem to have to_date, which makes me think that this is
> Drill UDF
> 4. Apparently, if you invoke to_date() with one argument in Drill: it runs.
>
> So there are two possibilities: we implemented to_date with one argument to
> be compatible with some other SQL engine, Hive ?
>or
>it is a bug and we should throw an error.
> You can use to_date with one argument in a simple query:
> {code}
> 0: jdbc:drill:schema=dfs> select to_date(c1 + interval '1' day) from t1 limit
> 1;
> +-+
> | EXPR$0|
> +-+
> | 2015-01-02 |
> +-+
> 1 row selected (0.242 seconds)
> {code}
> However, since return type is varbinary, joins, aggregations and CTAS are
> going to be problematic.
> Here is to_date use in join to illustrate this (c1 is a date column):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1, t2 where to_date(t1.c1) = t2.c2;
> Error: SYSTEM ERROR: DrillRuntimeException: Join only supports implicit casts
> between 1. Numeric data
> 2. Varchar, Varbinary data 3. Date, Timestamp data Left type: DATE, Right
> type: VAR16CHAR. Add explicit casts to avoid this error
> Fragment 0:0
> [Error Id: 66ac8248-56c5-401a-aa53-de90cb828bc4 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Since we don't support cast between varbinary and date, attempt to cast it
> results in:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1, t2 where cast(to_date(t1.c1) as
> date) = t2.c2;
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to
> materialize incoming schema. Errors:
> Error in expression at index -1. Error: Missing function implementation:
> [castBIGINT(VAR16CHAR-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--..
> Fragment 0:0
> [Error Id: deeb040a-f1d3-4ea0-8849-7ced29508576 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Same with CTAS:
> {code}
> 0: jdbc:drill:schema=dfs> create table x(a1) as select to_date(c1) from t1;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 10 |
> +---++
> 1 row selected (0.4 seconds)
> 0: jdbc:drill:schema=dfs> select * from x;
> +--+
> | a1 |
> +--+
> | [B@28b5395d |
> | [B@11c91d8c |
> | [B@2ab2db73 |
> | [B@446570eb |
> | [B@5fd87761 |
> | [B@7c85b26f |
> | [B@2d85d547 |
> | [B@2d753faa |
> | null |
> | [B@6ca6c936 |
> +--+
> 10 rows selected (0.183 seconds)
> 0: jdbc:drill:schema=dfs> select cast(a1 as date) from x;
> Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must
> be in the range [1,12]
> Fragment 0:0
> [Error Id: 71d8cd8f-6c88-4a13-9d24-b06ef52f6572 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3769) to_date function with one argument returns wrong data type
Victoria Markman created DRILL-3769:
---
Summary: to_date function with one argument returns wrong data type
Key: DRILL-3769
URL: https://issues.apache.org/jira/browse/DRILL-3769
Project: Apache Drill
Issue Type: Bug
Reporter: Victoria Markman
1. to_date function is not part of SQL standard according to my research
(checked ISO/IEC9075-2), so implementations of it may vary from database to
database (our implementation of to_date is different from Postgres, for example)
2. Our documentation only talks about to_date with 2 parameters: format and
actual string to be converted to date type
3. Calcite does not seem to have to_date, which makes me think that this is
Drill UDF
4. Apparently, if you invoke to_date() with one argument in Drill: it runs.
So there are two possibilities: we implemented to_date with one argument to be
compatible with some other SQL engine, Hive ?
or
it is a bug and we should throw an error.
You can use to_date with one argument in a simple query:
{code}
0: jdbc:drill:schema=dfs> select to_date(c1 + interval '1' day) from t1 limit 1;
+-+
| EXPR$0|
+-+
| 2015-01-02 |
+-+
1 row selected (0.242 seconds)
{code}
However, since return type is varbinary, joins, aggregations and CTAS are going
to be problematic.
Here is to_date use in join to illustrate this (c1 is a date column):
{code}
0: jdbc:drill:schema=dfs> select * from t1, t2 where to_date(t1.c1) = t2.c2;
Error: SYSTEM ERROR: DrillRuntimeException: Join only supports implicit casts
between 1. Numeric data
2. Varchar, Varbinary data 3. Date, Timestamp data Left type: DATE, Right
type: VAR16CHAR. Add explicit casts to avoid this error
Fragment 0:0
[Error Id: 66ac8248-56c5-401a-aa53-de90cb828bc4 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
Since we don't support cast between varbinary and date, attempt to cast it
results in:
{code}
0: jdbc:drill:schema=dfs> select * from t1, t2 where cast(to_date(t1.c1) as
date) = t2.c2;
Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize
incoming schema. Errors:
Error in expression at index -1. Error: Missing function implementation:
[castBIGINT(VAR16CHAR-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--..
Fragment 0:0
[Error Id: deeb040a-f1d3-4ea0-8849-7ced29508576 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
Same with CTAS:
{code}
0: jdbc:drill:schema=dfs> create table x(a1) as select to_date(c1) from t1;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 10 |
+---++
1 row selected (0.4 seconds)
0: jdbc:drill:schema=dfs> select * from x;
+--+
| a1 |
+--+
| [B@28b5395d |
| [B@11c91d8c |
| [B@2ab2db73 |
| [B@446570eb |
| [B@5fd87761 |
| [B@7c85b26f |
| [B@2d85d547 |
| [B@2d753faa |
| null |
| [B@6ca6c936 |
+--+
10 rows selected (0.183 seconds)
0: jdbc:drill:schema=dfs> select cast(a1 as date) from x;
Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must
be in the range [1,12]
Fragment 0:0
[Error Id: 71d8cd8f-6c88-4a13-9d24-b06ef52f6572 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3769) to_date function with one argument returns wrong data type
[
https://issues.apache.org/jira/browse/DRILL-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3769:
Component/s: Functions - Drill
> to_date function with one argument returns wrong data type
> --
>
> Key: DRILL-3769
> URL: https://issues.apache.org/jira/browse/DRILL-3769
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Victoria Markman
>
> 1. to_date function is not part of SQL standard according to my research
> (checked ISO/IEC9075-2), so implementations of it may vary from database to
> database (our implementation of to_date is different from Postgres, for
> example)
> 2. Our documentation only talks about to_date with 2 parameters: format and
> actual string to be converted to date type
> 3. Calcite does not seem to have to_date, which makes me think that this is
> Drill UDF
> 4. Apparently, if you invoke to_date() with one argument in Drill: it runs.
>
> So there are two possibilities: we implemented to_date with one argument to
> be compatible with some other SQL engine, Hive ?
>or
>it is a bug and we should throw an error.
> You can use to_date with one argument in a simple query:
> {code}
> 0: jdbc:drill:schema=dfs> select to_date(c1 + interval '1' day) from t1 limit
> 1;
> +-+
> | EXPR$0|
> +-+
> | 2015-01-02 |
> +-+
> 1 row selected (0.242 seconds)
> {code}
> However, since return type is varbinary, joins, aggregations and CTAS are
> going to be problematic.
> Here is to_date use in join to illustrate this (c1 is a date column):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1, t2 where to_date(t1.c1) = t2.c2;
> Error: SYSTEM ERROR: DrillRuntimeException: Join only supports implicit casts
> between 1. Numeric data
> 2. Varchar, Varbinary data 3. Date, Timestamp data Left type: DATE, Right
> type: VAR16CHAR. Add explicit casts to avoid this error
> Fragment 0:0
> [Error Id: 66ac8248-56c5-401a-aa53-de90cb828bc4 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Since we don't support cast between varbinary and date, attempt to cast it
> results in:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1, t2 where cast(to_date(t1.c1) as
> date) = t2.c2;
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to
> materialize incoming schema. Errors:
> Error in expression at index -1. Error: Missing function implementation:
> [castBIGINT(VAR16CHAR-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--..
> Fragment 0:0
> [Error Id: deeb040a-f1d3-4ea0-8849-7ced29508576 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Same with CTAS:
> {code}
> 0: jdbc:drill:schema=dfs> create table x(a1) as select to_date(c1) from t1;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 10 |
> +---++
> 1 row selected (0.4 seconds)
> 0: jdbc:drill:schema=dfs> select * from x;
> +--+
> | a1 |
> +--+
> | [B@28b5395d |
> | [B@11c91d8c |
> | [B@2ab2db73 |
> | [B@446570eb |
> | [B@5fd87761 |
> | [B@7c85b26f |
> | [B@2d85d547 |
> | [B@2d753faa |
> | null |
> | [B@6ca6c936 |
> +--+
> 10 rows selected (0.183 seconds)
> 0: jdbc:drill:schema=dfs> select cast(a1 as date) from x;
> Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must
> be in the range [1,12]
> Fragment 0:0
> [Error Id: 71d8cd8f-6c88-4a13-9d24-b06ef52f6572 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3382) CTAS with order by clause fails with IOOB exception
[
https://issues.apache.org/jira/browse/DRILL-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743769#comment-14743769
]
Victoria Markman commented on DRILL-3382:
-
Verified in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Test checked in under: Functional/Passing/ctas
> CTAS with order by clause fails with IOOB exception
> ---
>
> Key: DRILL-3382
> URL: https://issues.apache.org/jira/browse/DRILL-3382
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Parth Chandra
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3382-Fix-IOOB-error-for-CTAS-order-by-statemen.patch,
> 0001-DRILL-3382-Fix-IOOB-error-for-CTAS-order-by-statemen.patch
>
>
> The query :
> {panel}
> create table `lineitem__5` as select l_suppkey, l_partkey, l_linenumber from
> cp.`tpch/lineitem.parquet` l order by l_linenumber;
> {panel}
> fails with an IOOB exception
> Trace in log -
> {panel}
> [Error Id: 3351dcf3-032f-4d10-b2a4-c42959d0c06a on localhost:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IndexOutOfBoundsException: index (2) must be less than size (2)
> [Error Id: 3351dcf3-032f-4d10-b2a4-c42959d0c06a on localhost:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:737)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:839)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:781)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:783)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:892)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: Internal error: Error while
> applying rule ExpandConversionRule, args
> [rel#9013:AbstractConverter.PHYSICAL.SINGLETON([]).[2](input=rel#9011:Subset#7.PHYSICAL.ANY([]).[2],convention=PHYSICAL,DrillDistributionTraitDef=SINGLETON([]),sort=[2])]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ExpandConversionRule, args
> [rel#9013:AbstractConverter.PHYSICAL.SINGLETON([]).[2](input=rel#9011:Subset#7.PHYSICAL.ANY([]).[2],convention=PHYSICAL,DrillDistributionTraitDef=SINGLETON([]),sort=[2])]
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:260)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.CreateTableHandler.convertToPrel(CreateTableHandler.java:120)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.CreateTableHandler.getPlan(CreateTableHandler.java:99)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
> ~[drill-java-exec-1.1.0-SNAPSHOT.
[jira] [Closed] (DRILL-3382) CTAS with order by clause fails with IOOB exception
[
https://issues.apache.org/jira/browse/DRILL-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3382.
---
> CTAS with order by clause fails with IOOB exception
> ---
>
> Key: DRILL-3382
> URL: https://issues.apache.org/jira/browse/DRILL-3382
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Parth Chandra
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3382-Fix-IOOB-error-for-CTAS-order-by-statemen.patch,
> 0001-DRILL-3382-Fix-IOOB-error-for-CTAS-order-by-statemen.patch
>
>
> The query :
> {panel}
> create table `lineitem__5` as select l_suppkey, l_partkey, l_linenumber from
> cp.`tpch/lineitem.parquet` l order by l_linenumber;
> {panel}
> fails with an IOOB exception
> Trace in log -
> {panel}
> [Error Id: 3351dcf3-032f-4d10-b2a4-c42959d0c06a on localhost:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IndexOutOfBoundsException: index (2) must be less than size (2)
> [Error Id: 3351dcf3-032f-4d10-b2a4-c42959d0c06a on localhost:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:737)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:839)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:781)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:783)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:892)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: Internal error: Error while
> applying rule ExpandConversionRule, args
> [rel#9013:AbstractConverter.PHYSICAL.SINGLETON([]).[2](input=rel#9011:Subset#7.PHYSICAL.ANY([]).[2],convention=PHYSICAL,DrillDistributionTraitDef=SINGLETON([]),sort=[2])]
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ExpandConversionRule, args
> [rel#9013:AbstractConverter.PHYSICAL.SINGLETON([]).[2](input=rel#9011:Subset#7.PHYSICAL.ANY([]).[2],convention=PHYSICAL,DrillDistributionTraitDef=SINGLETON([]),sort=[2])]
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316)
> ~[calcite-core-1.1.0-drill-r9.jar:1.1.0-drill-r9]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPrel(DefaultSqlHandler.java:260)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.CreateTableHandler.convertToPrel(CreateTableHandler.java:120)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.CreateTableHandler.getPlan(CreateTableHandler.java:99)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
> ~[drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:903)
> [drill-java-exec-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.wor
[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743789#comment-14743789
]
Victoria Markman commented on DRILL-2748:
-
Jinfeng, this is checked in already, correct ?
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2418) Memory leak during execution if comparison function is not found
[
https://issues.apache.org/jira/browse/DRILL-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744390#comment-14744390
]
Victoria Markman commented on DRILL-2418:
-
The original error happened much later during execution. After we started
throwing this new error message, memory leak is not reproducible any more.
I automated not supported implicit cast cases (under
Functional/Passing/joins/implicit_cast_not_supported) and ran these in a loop
of 20 iterations with 10 concurrent queries and I don't see memory leak any
more.
{code}
heap(b) direct(b) jvm_direct(b)
2442800296 117991561509996862
2207849320 117989411493219626
1968600792 117989411493219598
1737329064 117989411493219582
1503715872 117989411493219558
1282546296 117989411493219538
1050979952 117989411493203108
3357847152 117991561493203340
3120888640 117989411493203308
2885817664 117989411493203288
2654219216 117989411493203268
2421491408 117991561493203236
2193786112 117989411493203220
1957000352 117989411493186745
1725191640 117989411493186725
1500829160 117989411493186701
1282839416 117989411493186689
1061937448 117989411493186661
3366707496 117989411493186893
3131793720 117989411493186865
2901312272 117989411493186849
{code}
> Memory leak during execution if comparison function is not found
>
>
> Key: DRILL-2418
> URL: https://issues.apache.org/jira/browse/DRILL-2418
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 0.8.0
>Reporter: Victoria Markman
>Assignee: Chris Westin
> Fix For: 1.2.0
>
> Attachments: cast_tbl_1.parquet, cast_tbl_2.parquet,
> not_supported_cast.txt
>
>
> While testing implicit cast during join, I ran into an issue where if you run
> a query that throws an exception during execution, eventually, if you run
> enough of those, drill will run out of memory.
> Here is a query example:
> {code}
> select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.c_float = b.c_time
> failed: RemoteRpcException: Failure while running fragment., Failure finding
> function that runtime code generation expected. Signature:
> compare_to_nulls_high( TIME:OPTIONAL, FLOAT4:OPTIONAL ) returns INT:REQUIRED
> [ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
> [ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
> Test_Failed: 2015/03/10 18:34:15.0015 - Failed to execute.
> {code}
> If you set planner.slice_target to 1, you hit out of memory after about ~40
> or so of such failures on my cluster.
> {code}
> select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.d38 = b.c_double
> Query failed: OutOfMemoryException: You attempted to create a new child
> allocator with initial reservation 300 but only 916199 bytes of memory
> were available.
> {code}
> From the drillbit.log
> {code}
> 2015-03-10 18:34:34,588 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out
> of 1 using 1 threads. Time: 1ms total, 1.190007ms avg, 1ms max.
> 2015-03-10 18:34:34,591 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out
> of 1 using 1 threads. Time: 0ms total, 0.953679ms avg, 0ms max.
> 2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
> atsqa4-136.qa.lab. Skipping affinity to that host.
> 2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps:
> Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.609586ms avg, 1ms max.
> 2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
> atsqa4-136.qa.lab. Skipping affinity to that host.
> 2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps:
> Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.270340ms avg, 1ms max.
> 2015-03-10 18:34:34,684 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING -->
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: Failure while getting memory all
[jira] [Commented] (DRILL-2549) Implicit Cast between integer and varchar columns in a join should work
[
https://issues.apache.org/jira/browse/DRILL-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744395#comment-14744395
]
Victoria Markman commented on DRILL-2549:
-
[~rkins] This issue might be already fixed (DRILL-2418), can you please check
it out ? Thanks !
> Implicit Cast between integer and varchar columns in a join should work
> ---
>
> Key: DRILL-2549
> URL: https://issues.apache.org/jira/browse/DRILL-2549
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Rahul Challapalli
>Assignee: Mehant Baid
> Fix For: Future
>
> Attachments: error.log, fewtypes.tbl, fewtypes_null.parquet
>
>
> git.commit.id.abbrev=f1b59ed
> I attached 2 files which contain the same values
> The below query is not doing an implicit cast between a varchar and an integer
> {code}
> select count(*) from dfs.`cross-sources`.`fewtypes_null.parquet` p
> inner join dfs.`cross-sources`.`fewtypes_null.tbl` o
> on p.int_col=o.columns[0];
> Query failed: RemoteRpcException: Failure while running fragment., Failure
> finding function that runtime code generation expected. Signature:
> compare_to_nulls_high( INT:OPTIONAL, VARCHAR:OPTIONAL ) returns INT:REQUIRED
> [ 08df7d57-c436-4ff8-9a31-48368ba05531 on qa-node190.qa.lab:31010 ]
> [ 08df7d57-c436-4ff8-9a31-48368ba05531 on qa-node190.qa.lab:31010 ]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3412) Projections are not getting push down below Window operator
[
https://issues.apache.org/jira/browse/DRILL-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744588#comment-14744588
]
Victoria Markman commented on DRILL-3412:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Added bunch of test cases under:
Functional/Passing/window_functions/optimization
> Projections are not getting push down below Window operator
> ---
>
> Key: DRILL-3412
> URL: https://issues.apache.org/jira/browse/DRILL-3412
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Aman Sinha
>Assignee: Sean Hsuan-Yi Chu
>Priority: Blocker
> Labels: window_function
> Fix For: 1.2.0
>
>
> The plan below shows that the 'star' column is being produced by the Scan and
> subsequent Project. This indicates projection pushdown is not working as
> desired when window function is present. The query produces correct results.
> {code}
> explain plan for select min(n_nationkey) over (partition by n_regionkey) from
> cp.`tpch/nation.parquet` ;
> 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02Project(w0$o0=[$3])
> 00-03 Window(window#0=[window(partition {2} order by [] range
> between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [MIN($1)])])
> 00-04SelectionVectorRemover
> 00-05 Sort(sort0=[$2], dir0=[ASC])
> 00-06Project(T1¦¦*=[$0], n_nationkey=[$1], n_regionkey=[$2])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
> selectionRoot=/tpch/nation.parquet, numFiles=1, columns=[`*`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3412) Projections are not getting push down below Window operator
[
https://issues.apache.org/jira/browse/DRILL-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3412.
---
> Projections are not getting push down below Window operator
> ---
>
> Key: DRILL-3412
> URL: https://issues.apache.org/jira/browse/DRILL-3412
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Aman Sinha
>Assignee: Sean Hsuan-Yi Chu
>Priority: Blocker
> Labels: window_function
> Fix For: 1.2.0
>
>
> The plan below shows that the 'star' column is being produced by the Scan and
> subsequent Project. This indicates projection pushdown is not working as
> desired when window function is present. The query produces correct results.
> {code}
> explain plan for select min(n_nationkey) over (partition by n_regionkey) from
> cp.`tpch/nation.parquet` ;
> 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02Project(w0$o0=[$3])
> 00-03 Window(window#0=[window(partition {2} order by [] range
> between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [MIN($1)])])
> 00-04SelectionVectorRemover
> 00-05 Sort(sort0=[$2], dir0=[ASC])
> 00-06Project(T1¦¦*=[$0], n_nationkey=[$1], n_regionkey=[$2])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
> selectionRoot=/tpch/nation.parquet, numFiles=1, columns=[`*`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2724) Implicit cast test fails in Union All query (reports type mismatch)
[
https://issues.apache.org/jira/browse/DRILL-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745588#comment-14745588
]
Victoria Markman commented on DRILL-2724:
-
Closing, as this is not a bug, but expected behavior.
> Implicit cast test fails in Union All query (reports type mismatch)
> ---
>
> Key: DRILL-2724
> URL: https://issues.apache.org/jira/browse/DRILL-2724
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
> Environment: 4 node cluster on CentOS
> | 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580: Exit early from
> HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53 EDT
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Test that performs implicit cast in a Union All query fails. Each CSV file
> has only numeric data. Test was performed on 4 node cluster.
> {code}
> With casting, assuming implicit casting would work. Data in each of the csv
> files is numeric.
> 0: jdbc:drill:> select * from (select cast(columns[0] as bigint) from
> `bgint_f.csv` union all select cast(columns[0] as char(2)) from `char_f.csv`
> union all select cast(columns[0] as double) from `dbl_f.csv` union all select
> cast(columns[0] as float) from `float_f.csv` union all select cast(columns[0]
> as int) from `int_f.csv` union all select cast(columns[0] as varchar(100))
> from `vchar_f.csv`) order by EXPR$0;
> Query failed: SqlValidatorException: Type mismatch in column 1 of UNION ALL
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> explain plan did not return the physical plan
> 0: jdbc:drill:> explain plan for select * from (select cast(columns[0] as
> bigint) from `bgint_f.csv` union all select cast(columns[0] as char(2)) from
> `char_f.csv` union all select cast(columns[0] as double) from `dbl_f.csv`
> union all select cast(columns[0] as float) from `float_f.csv` union all
> select cast(columns[0] as int) from `int_f.csv` union all select
> cast(columns[0] as varchar(100)) from `vchar_f.csv`) order by EXPR$0;
> Query failed: SqlValidatorException: Type mismatch in column 1 of UNION ALL
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> Stack trace from drillbit.log
> 2015-04-08 21:30:44,553 [2ada61fa-8207-279a-97fd-b40631cdb151:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - Error
> 00ab460b-d2be-441e-8baa-0a8ff474769e: SqlValidatorException: Type mismatch in
> column 1 of UNION ALL
> org.apache.drill.exec.planner.sql.QueryInputException: Failure validating
> SQL. org.eigenbase.util.EigenbaseContextException: From line 1, column 23 to
> line 1, column 48: Type mismatch in column 1 of UNION ALL
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:147)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_75]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_75]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
> Caused by: net.hydromatic.optiq.tools.ValidationException:
> org.eigenbase.util.EigenbaseContextException: From line 1, column 23 to line
> 1, column 48: Type mismatch in column 1 of UNION ALL
> at
> net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:176)
> ~[optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:157)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:133)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> ... 5 common frames omitted
> Caused by: org.eigenbase.util.EigenbaseContextException: From line 1, column
> 23 to line 1, column 48: Type mismatch in column 1 of UNION ALL
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.7.0_75]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
[jira] [Closed] (DRILL-2724) Implicit cast test fails in Union All query (reports type mismatch)
[
https://issues.apache.org/jira/browse/DRILL-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2724.
---
> Implicit cast test fails in Union All query (reports type mismatch)
> ---
>
> Key: DRILL-2724
> URL: https://issues.apache.org/jira/browse/DRILL-2724
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
> Environment: 4 node cluster on CentOS
> | 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580: Exit early from
> HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53 EDT
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Test that performs implicit cast in a Union All query fails. Each CSV file
> has only numeric data. Test was performed on 4 node cluster.
> {code}
> With casting, assuming implicit casting would work. Data in each of the csv
> files is numeric.
> 0: jdbc:drill:> select * from (select cast(columns[0] as bigint) from
> `bgint_f.csv` union all select cast(columns[0] as char(2)) from `char_f.csv`
> union all select cast(columns[0] as double) from `dbl_f.csv` union all select
> cast(columns[0] as float) from `float_f.csv` union all select cast(columns[0]
> as int) from `int_f.csv` union all select cast(columns[0] as varchar(100))
> from `vchar_f.csv`) order by EXPR$0;
> Query failed: SqlValidatorException: Type mismatch in column 1 of UNION ALL
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> explain plan did not return the physical plan
> 0: jdbc:drill:> explain plan for select * from (select cast(columns[0] as
> bigint) from `bgint_f.csv` union all select cast(columns[0] as char(2)) from
> `char_f.csv` union all select cast(columns[0] as double) from `dbl_f.csv`
> union all select cast(columns[0] as float) from `float_f.csv` union all
> select cast(columns[0] as int) from `int_f.csv` union all select
> cast(columns[0] as varchar(100)) from `vchar_f.csv`) order by EXPR$0;
> Query failed: SqlValidatorException: Type mismatch in column 1 of UNION ALL
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> Stack trace from drillbit.log
> 2015-04-08 21:30:44,553 [2ada61fa-8207-279a-97fd-b40631cdb151:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - Error
> 00ab460b-d2be-441e-8baa-0a8ff474769e: SqlValidatorException: Type mismatch in
> column 1 of UNION ALL
> org.apache.drill.exec.planner.sql.QueryInputException: Failure validating
> SQL. org.eigenbase.util.EigenbaseContextException: From line 1, column 23 to
> line 1, column 48: Type mismatch in column 1 of UNION ALL
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:147)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_75]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_75]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
> Caused by: net.hydromatic.optiq.tools.ValidationException:
> org.eigenbase.util.EigenbaseContextException: From line 1, column 23 to line
> 1, column 48: Type mismatch in column 1 of UNION ALL
> at
> net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:176)
> ~[optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:157)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:133)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> ... 5 common frames omitted
> Caused by: org.eigenbase.util.EigenbaseContextException: From line 1, column
> 23 to line 1, column 48: Type mismatch in column 1 of UNION ALL
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[na:1.7.0_75]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> ~[na:1.7.0_75]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAcc
[jira] [Commented] (DRILL-2688) Use of ORDER BY on right side of Union All results in SqlValidatorException
[
https://issues.apache.org/jira/browse/DRILL-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745608#comment-14745608
]
Victoria Markman commented on DRILL-2688:
-
ISO/IEC 9075-2:2011(E) (6.41 ) states: "b) QE shall
not immediately contain an "
Example from postgres:
{code}
postgres=# select a1
postgres-# fromt1
postgres-# where a1 = 1
postgres-# UNION ALL
postgres-# select a2
postgres-# fromt2
postgres-# where a2 is not null
postgres-# group by a1
postgres-# order by a1;
ERROR: column "a1" does not exist
LINE 8: group by a1
^
HINT: There is a column named "a1" in table "*SELECT* 1", but it cannot be
referenced from this part of the query.
{code}
If you want to sort result of union all (drill 1.2.0)
{code}
0: jdbc:drill:schema=dfs> select * from
. . . . . . . . . . . . > (
. . . . . . . . . . . . > select a1
. . . . . . . . . . . . > fromt1
. . . . . . . . . . . . > where a1 = 1
. . . . . . . . . . . . > UNION ALL
. . . . . . . . . . . . > select a2
. . . . . . . . . . . . > fromt2
. . . . . . . . . . . . > where a2 is not null
. . . . . . . . . . . . > ) as dt
. . . . . . . . . . . . > group by a1
. . . . . . . . . . . . > order by a1;
+-+
| a1 |
+-+
| 0 |
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
| 7 |
| 8 |
| 9 |
+-+
10 rows selected (0.448 seconds)
{code}
> Use of ORDER BY on right side of Union All results in SqlValidatorException
> ---
>
> Key: DRILL-2688
> URL: https://issues.apache.org/jira/browse/DRILL-2688
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
> Environment: 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580:
> Exit early from HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53
> EDT
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Minor
> Fix For: 1.2.0
>
>
> Order by in the right leg of Union All results in SqlValidatorException.
> Executing the query individually (the query that is on right side of Union
> All) gives expected results and no Exceptions are seen. However, executing
> the query as part of the right leg of Union All results in Exception.
> Tests were executed on 4 node cluster on CentOS. Both sides of Union All got
> input from CSV files.
> The query below returns correct results when executed individually. (this is
> the same query used on right side of Union All query)
> {code}
> 0: jdbc:drill:> select columns[0] from `testWindow.csv` t2 where
> t2.columns[0] is not null group by columns[0] order by columns[0];
> ++
> | EXPR$0 |
> ++
> | 10 |
> | 100|
> | 113|
> | 119|
> | 2 |
> | 50 |
> | 55 |
> | 57 |
> | 61 |
> | 67 |
> | 89 |
> ++
> 11 rows selected (0.265 seconds)
> {code}
> Note that the Union All query works when there is no order by in the right
> leg.
> {code}
> 0: jdbc:drill:> select columns[0] from `employee.csv` t1 where
> t1.columns[0]=1 union all select columns[0] from `testWindow.csv` t2 where
> t2.columns[0] is not null group by columns[0];
> ++
> | EXPR$0 |
> ++
> | 1 |
> | 100|
> | 10 |
> | 2 |
> | 50 |
> | 55 |
> | 67 |
> | 113|
> | 119|
> | 89 |
> | 57 |
> | 61 |
> ++
> 12 rows selected (0.175 seconds)
> {code}
> In this case we do use an order by on the right side of Union All, which
> results in SqlValidatorException. Same query as above, except that we have an
> order by clause in the query on the right side of Union All.
> {code}
> 0: jdbc:drill:> select columns[0] from `employee.csv` t1 where
> t1.columns[0]=1 union all select columns[0] from `testWindow.csv` t2 where
> t2.columns[0] is not null group by columns[0] order by columns[0];
> Query failed: SqlValidatorException: Column 'columns' not found in any table
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> 2015-04-04 04:31:22,882 [2ae096e5-4085-de95-5642-ae10d320a8f7:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - Error
> c6a60f71-c959-4ae5-ba8c-23b9c7b6bb7e: SqlValidatorException: Column 'columns'
> not found in any table
> org.apache.drill.exec.planner.sql.QueryInputException: Failure validating
> SQL. org.eigenbase.util.EigenbaseContextException: From line 1, column 178 to
> line 1, column 184: Column 'columns' not found in any table
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan
[jira] [Closed] (DRILL-2688) Use of ORDER BY on right side of Union All results in SqlValidatorException
[
https://issues.apache.org/jira/browse/DRILL-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2688.
---
> Use of ORDER BY on right side of Union All results in SqlValidatorException
> ---
>
> Key: DRILL-2688
> URL: https://issues.apache.org/jira/browse/DRILL-2688
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
> Environment: 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580:
> Exit early from HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53
> EDT
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Minor
> Fix For: 1.2.0
>
>
> Order by in the right leg of Union All results in SqlValidatorException.
> Executing the query individually (the query that is on right side of Union
> All) gives expected results and no Exceptions are seen. However, executing
> the query as part of the right leg of Union All results in Exception.
> Tests were executed on 4 node cluster on CentOS. Both sides of Union All got
> input from CSV files.
> The query below returns correct results when executed individually. (this is
> the same query used on right side of Union All query)
> {code}
> 0: jdbc:drill:> select columns[0] from `testWindow.csv` t2 where
> t2.columns[0] is not null group by columns[0] order by columns[0];
> ++
> | EXPR$0 |
> ++
> | 10 |
> | 100|
> | 113|
> | 119|
> | 2 |
> | 50 |
> | 55 |
> | 57 |
> | 61 |
> | 67 |
> | 89 |
> ++
> 11 rows selected (0.265 seconds)
> {code}
> Note that the Union All query works when there is no order by in the right
> leg.
> {code}
> 0: jdbc:drill:> select columns[0] from `employee.csv` t1 where
> t1.columns[0]=1 union all select columns[0] from `testWindow.csv` t2 where
> t2.columns[0] is not null group by columns[0];
> ++
> | EXPR$0 |
> ++
> | 1 |
> | 100|
> | 10 |
> | 2 |
> | 50 |
> | 55 |
> | 67 |
> | 113|
> | 119|
> | 89 |
> | 57 |
> | 61 |
> ++
> 12 rows selected (0.175 seconds)
> {code}
> In this case we do use an order by on the right side of Union All, which
> results in SqlValidatorException. Same query as above, except that we have an
> order by clause in the query on the right side of Union All.
> {code}
> 0: jdbc:drill:> select columns[0] from `employee.csv` t1 where
> t1.columns[0]=1 union all select columns[0] from `testWindow.csv` t2 where
> t2.columns[0] is not null group by columns[0] order by columns[0];
> Query failed: SqlValidatorException: Column 'columns' not found in any table
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> 2015-04-04 04:31:22,882 [2ae096e5-4085-de95-5642-ae10d320a8f7:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - Error
> c6a60f71-c959-4ae5-ba8c-23b9c7b6bb7e: SqlValidatorException: Column 'columns'
> not found in any table
> org.apache.drill.exec.planner.sql.QueryInputException: Failure validating
> SQL. org.eigenbase.util.EigenbaseContextException: From line 1, column 178 to
> line 1, column 184: Column 'columns' not found in any table
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:147)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_75]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_75]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
> Caused by: net.hydromatic.optiq.tools.ValidationException:
> org.eigenbase.util.EigenbaseContextException: From line 1, column 178 to line
> 1, column 184: Column 'columns' not found in any table
> at
> net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:176)
> ~[optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.validateNode(DefaultSqlHandler.java:157)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:133)
> ~[d
[jira] [Commented] (DRILL-2217) Trying to flatten an empty list should return an empty result
[
https://issues.apache.org/jira/browse/DRILL-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745802#comment-14745802
]
Victoria Markman commented on DRILL-2217:
-
With drill 1.2.0 I see result below and I'm not sure if it is correct ...
{code}
0: jdbc:drill:schema=dfs> select empty from `data1.json`;
+--+
|empty |
+--+
| [[],[null]] |
| [[],[null]] |
+--+
2 rows selected (0.167 seconds)
0: jdbc:drill:schema=dfs> select flatten(empty) from `data1.json`;
+-+
| EXPR$0 |
+-+
| null|
| null|
| null|
| null|
+-+
4 rows selected (0.174 seconds)
{code}
> Trying to flatten an empty list should return an empty result
> -
>
> Key: DRILL-2217
> URL: https://issues.apache.org/jira/browse/DRILL-2217
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Rahul Challapalli
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
> Attachments: error.log
>
>
> git.commit.id.abbrev=3d863b5
> Data Set :
> {code}
> {"empty":[[],[[]]]}
> {code}
> Query :
> {code}
> select flatten(empty) from `data1.json`;
> Query failed: RemoteRpcException: Failure while running fragment.[
> 1b3123d9-92bc-45d5-bef8-b5f1be9def07 on qa-node191.qa.lab:31010 ]
> [ 1b3123d9-92bc-45d5-bef8-b5f1be9def07 on qa-node191.qa.lab:31010 ]
> {code}
> I also attached the error from the logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (DRILL-2217) Trying to flatten an empty list should return an empty result
[
https://issues.apache.org/jira/browse/DRILL-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2217:
Comment: was deleted
(was: With drill 1.2.0 I see result below and I'm not sure if it is correct ...
{code}
0: jdbc:drill:schema=dfs> select empty from `data1.json`;
+--+
|empty |
+--+
| [[],[null]] |
| [[],[null]] |
+--+
2 rows selected (0.167 seconds)
0: jdbc:drill:schema=dfs> select flatten(empty) from `data1.json`;
+-+
| EXPR$0 |
+-+
| null|
| null|
| null|
| null|
+-+
4 rows selected (0.174 seconds)
{code})
> Trying to flatten an empty list should return an empty result
> -
>
> Key: DRILL-2217
> URL: https://issues.apache.org/jira/browse/DRILL-2217
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Rahul Challapalli
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
> Attachments: error.log
>
>
> git.commit.id.abbrev=3d863b5
> Data Set :
> {code}
> {"empty":[[],[[]]]}
> {code}
> Query :
> {code}
> select flatten(empty) from `data1.json`;
> Query failed: RemoteRpcException: Failure while running fragment.[
> 1b3123d9-92bc-45d5-bef8-b5f1be9def07 on qa-node191.qa.lab:31010 ]
> [ 1b3123d9-92bc-45d5-bef8-b5f1be9def07 on qa-node191.qa.lab:31010 ]
> {code}
> I also attached the error from the logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-2777) CTAS, order by and flatten of repeated list result in ExpandConversionRule error
[
https://issues.apache.org/jira/browse/DRILL-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2777.
---
> CTAS, order by and flatten of repeated list result in ExpandConversionRule
> error
>
>
> Key: DRILL-2777
> URL: https://issues.apache.org/jira/browse/DRILL-2777
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: data.json, error.log
>
>
> git.commit.id.abbrev=f066786
> The below query results in an error :
> {code}
> create table dfs.ctas_flatten.`test` as select uid, flatten(lst_lst) lst,
> flatten(lst_lst[0]) from `data1.json` order by flatten(lst_lst[0]);
> Query failed: IndexOutOfBoundsException: index (2) must be less than size (2)
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> The logs indicate an assertion error :
> {code}
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ExpandConversionRule, args
> [rel#1757858:AbstractConverter.PHYSICAL.SINGLETON([]).[2](child=rel#1757856:Subset#8.PHYSICAL.ANY([]).[2],convention=PHYSICAL,DrillDistributionTraitDef=SINGLETON([]),sort=[2])]
> {code}
> The query itself does not have any issues. Also the below variants work :
> {code}
> 1. No flatten of repeated list in the select clause
> create table dfs.ctas_flatten.`test` as select uid, flatten(lst_lst[0]) from
> `data1.json` order by flatten(lst_lst[0]);
> 2. Removing the flatten from the order by. (Ordering by a scalar has no
> issues)
> create table dfs.ctas_flatten.`test` as select uid, flatten(lst_lst) lst,
> flatten(lst_lst[0]) from `data1.json`;
> {code}
> Attached the data set and the complete error log file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2777) CTAS, order by and flatten of repeated list result in ExpandConversionRule error
[
https://issues.apache.org/jira/browse/DRILL-2777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745830#comment-14745830
]
Victoria Markman commented on DRILL-2777:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Test added to: Functional/Passing/flatten_operators/negative
> CTAS, order by and flatten of repeated list result in ExpandConversionRule
> error
>
>
> Key: DRILL-2777
> URL: https://issues.apache.org/jira/browse/DRILL-2777
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Rahul Challapalli
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: data.json, error.log
>
>
> git.commit.id.abbrev=f066786
> The below query results in an error :
> {code}
> create table dfs.ctas_flatten.`test` as select uid, flatten(lst_lst) lst,
> flatten(lst_lst[0]) from `data1.json` order by flatten(lst_lst[0]);
> Query failed: IndexOutOfBoundsException: index (2) must be less than size (2)
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> The logs indicate an assertion error :
> {code}
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ExpandConversionRule, args
> [rel#1757858:AbstractConverter.PHYSICAL.SINGLETON([]).[2](child=rel#1757856:Subset#8.PHYSICAL.ANY([]).[2],convention=PHYSICAL,DrillDistributionTraitDef=SINGLETON([]),sort=[2])]
> {code}
> The query itself does not have any issues. Also the below variants work :
> {code}
> 1. No flatten of repeated list in the select clause
> create table dfs.ctas_flatten.`test` as select uid, flatten(lst_lst[0]) from
> `data1.json` order by flatten(lst_lst[0]);
> 2. Removing the flatten from the order by. (Ordering by a scalar has no
> issues)
> create table dfs.ctas_flatten.`test` as select uid, flatten(lst_lst) lst,
> flatten(lst_lst[0]) from `data1.json`;
> {code}
> Attached the data set and the complete error log file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3700) Exception in a query with multiple fist_value window functions with different partitions
[
https://issues.apache.org/jira/browse/DRILL-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745848#comment-14745848
]
Victoria Markman commented on DRILL-3700:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Test added to: Functional/Passing/window_functions/bugs/drill-3700.*
> Exception in a query with multiple fist_value window functions with different
> partitions
>
>
> Key: DRILL-3700
> URL: https://issues.apache.org/jira/browse/DRILL-3700
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
> Labels: window_function
> Fix For: 1.2.0
>
> Attachments: j6.tar
>
>
> {code}
> alter session set `planner.slice_target` = 1;
> select
> first_value(c_integer) over(order by c_integer nulls first),
> first_value(c_bigint) over(partition by c_time),
> first_value(c_integer) over (partition by c_time),
> first_value(c_bigint) over(partition by c_time order by c_date),
> first_value(c_integer) over (partition by c_time order by c_date)
> from
> j6;
> {code}
> drillbit.log
> {code}
> 2015-08-24 15:43:14,157 [2a24c474-67f1-2359-d25d-f4e9c7f9e98b:frag:3:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> OversizedAllocationException: Unable to expand the buffer. Max allowed buffer
> size is reached.
> Fragment 3:0
> [Error Id: b5e13008-b128-4c39-b25d-9e1529296099 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> OversizedAllocationException: Unable to expand the buffer. Max allowed buffer
> size is reached.
> Fragment 3:0
> [Error Id: b5e13008-b128-4c39-b25d-9e1529296099 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.exec.exception.OversizedAllocationException:
> Unable to expand the buffer. Max allowed buffer size is reached.
> at
> org.apache.drill.exec.vector.BigIntVector.reAlloc(BigIntVector.java:205)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.BigIntVector$Mutator.setSafe(BigIntVector.java:403)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableBigIntVector$Mutator.setSafe(NullableBigIntVector.java:481)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.copyFirstValueToInternal(DefaultFrameTemplate.java:243)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.newPartition(DefaultFrameTemplate.java:140)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.doWork(DefaultFrameTemplate.java:108)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.window.WindowFrameRecordBatch.innerNext(WindowFrameRecordBatch.java:157)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:95)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.dri
[jira] [Closed] (DRILL-3700) Exception in a query with multiple fist_value window functions with different partitions
[
https://issues.apache.org/jira/browse/DRILL-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3700.
---
> Exception in a query with multiple fist_value window functions with different
> partitions
>
>
> Key: DRILL-3700
> URL: https://issues.apache.org/jira/browse/DRILL-3700
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.2.0
>
> Attachments: j6.tar
>
>
> {code}
> alter session set `planner.slice_target` = 1;
> select
> first_value(c_integer) over(order by c_integer nulls first),
> first_value(c_bigint) over(partition by c_time),
> first_value(c_integer) over (partition by c_time),
> first_value(c_bigint) over(partition by c_time order by c_date),
> first_value(c_integer) over (partition by c_time order by c_date)
> from
> j6;
> {code}
> drillbit.log
> {code}
> 2015-08-24 15:43:14,157 [2a24c474-67f1-2359-d25d-f4e9c7f9e98b:frag:3:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> OversizedAllocationException: Unable to expand the buffer. Max allowed buffer
> size is reached.
> Fragment 3:0
> [Error Id: b5e13008-b128-4c39-b25d-9e1529296099 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> OversizedAllocationException: Unable to expand the buffer. Max allowed buffer
> size is reached.
> Fragment 3:0
> [Error Id: b5e13008-b128-4c39-b25d-9e1529296099 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.exec.exception.OversizedAllocationException:
> Unable to expand the buffer. Max allowed buffer size is reached.
> at
> org.apache.drill.exec.vector.BigIntVector.reAlloc(BigIntVector.java:205)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.BigIntVector$Mutator.setSafe(BigIntVector.java:403)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableBigIntVector$Mutator.setSafe(NullableBigIntVector.java:481)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.copyFirstValueToInternal(DefaultFrameTemplate.java:243)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.newPartition(DefaultFrameTemplate.java:140)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.doWork(DefaultFrameTemplate.java:108)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.window.WindowFrameRecordBatch.innerNext(WindowFrameRecordBatch.java:157)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:95)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:129)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(
[jira] [Updated] (DRILL-3700) Exception in a query with multiple fist_value window functions with different partitions
[
https://issues.apache.org/jira/browse/DRILL-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3700:
Assignee: Deneche A. Hakim (was: Victoria Markman)
> Exception in a query with multiple fist_value window functions with different
> partitions
>
>
> Key: DRILL-3700
> URL: https://issues.apache.org/jira/browse/DRILL-3700
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.2.0
>
> Attachments: j6.tar
>
>
> {code}
> alter session set `planner.slice_target` = 1;
> select
> first_value(c_integer) over(order by c_integer nulls first),
> first_value(c_bigint) over(partition by c_time),
> first_value(c_integer) over (partition by c_time),
> first_value(c_bigint) over(partition by c_time order by c_date),
> first_value(c_integer) over (partition by c_time order by c_date)
> from
> j6;
> {code}
> drillbit.log
> {code}
> 2015-08-24 15:43:14,157 [2a24c474-67f1-2359-d25d-f4e9c7f9e98b:frag:3:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> OversizedAllocationException: Unable to expand the buffer. Max allowed buffer
> size is reached.
> Fragment 3:0
> [Error Id: b5e13008-b128-4c39-b25d-9e1529296099 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> OversizedAllocationException: Unable to expand the buffer. Max allowed buffer
> size is reached.
> Fragment 3:0
> [Error Id: b5e13008-b128-4c39-b25d-9e1529296099 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.exec.exception.OversizedAllocationException:
> Unable to expand the buffer. Max allowed buffer size is reached.
> at
> org.apache.drill.exec.vector.BigIntVector.reAlloc(BigIntVector.java:205)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.BigIntVector$Mutator.setSafe(BigIntVector.java:403)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableBigIntVector$Mutator.setSafe(NullableBigIntVector.java:481)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.copyFirstValueToInternal(DefaultFrameTemplate.java:243)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.newPartition(DefaultFrameTemplate.java:140)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.WindowFramerGen141.doWork(DefaultFrameTemplate.java:108)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.window.WindowFrameRecordBatch.innerNext(WindowFrameRecordBatch.java:157)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:95)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:129)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
[jira] [Commented] (DRILL-3189) Disable ALLOW PARTIAL/DISALLOW PARTIAL in window function grammar
[
https://issues.apache.org/jira/browse/DRILL-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745884#comment-14745884
]
Victoria Markman commented on DRILL-3189:
-
Since we decided not to support "ROWS" syntax for now, it's not possible to
verify this fix.
> Disable ALLOW PARTIAL/DISALLOW PARTIAL in window function grammar
> -
>
> Key: DRILL-3189
> URL: https://issues.apache.org/jira/browse/DRILL-3189
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> It does not seem to be implemented on the drill side. Looks like Calcite
> specific grammar. Don't see it SQL Standard.
> Looks like wrong result:
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following disallow partial) from t2 order by
> a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | null|
> | 1 | null|
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | null|
> | 4 | null|
> | 5 | null|
> | 6 | null|
> | 7 | 14 |
> | 7 | 14 |
> | 8 | null|
> | 9 | null|
> +-+-+
> 13 rows selected (0.213 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following allow partial) from t2 order by a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | 0 |
> | 1 | 1 |
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | 3 |
> | 4 | 4 |
> | 5 | 5 |
> | 6 | 6 |
> | 7 | 14 |
> | 7 | 14 |
> | 8 | 8 |
> | 9 | 9 |
> +-+-+
> 13 rows selected (0.208 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> disallow partial) from t2 order by a2;
> Error: PARSE ERROR: From line 1, column 53 to line 1, column 68: Cannot use
> DISALLOW PARTIAL with window based on RANGE
> [Error Id: 984c4b81-9eb0-401d-b36a-9580640b4a78 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3189) Disable ALLOW PARTIAL/DISALLOW PARTIAL in window function grammar
[
https://issues.apache.org/jira/browse/DRILL-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3189.
---
> Disable ALLOW PARTIAL/DISALLOW PARTIAL in window function grammar
> -
>
> Key: DRILL-3189
> URL: https://issues.apache.org/jira/browse/DRILL-3189
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> It does not seem to be implemented on the drill side. Looks like Calcite
> specific grammar. Don't see it SQL Standard.
> Looks like wrong result:
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following disallow partial) from t2 order by
> a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | null|
> | 1 | null|
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | null|
> | 4 | null|
> | 5 | null|
> | 6 | null|
> | 7 | 14 |
> | 7 | 14 |
> | 8 | null|
> | 9 | null|
> +-+-+
> 13 rows selected (0.213 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following allow partial) from t2 order by a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | 0 |
> | 1 | 1 |
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | 3 |
> | 4 | 4 |
> | 5 | 5 |
> | 6 | 6 |
> | 7 | 14 |
> | 7 | 14 |
> | 8 | 8 |
> | 9 | 9 |
> +-+-+
> 13 rows selected (0.208 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> disallow partial) from t2 order by a2;
> Error: PARSE ERROR: From line 1, column 53 to line 1, column 68: Cannot use
> DISALLOW PARTIAL with window based on RANGE
> [Error Id: 984c4b81-9eb0-401d-b36a-9580640b4a78 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3189) Disable ALLOW PARTIAL/DISALLOW PARTIAL in window function grammar
[
https://issues.apache.org/jira/browse/DRILL-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745884#comment-14745884
]
Victoria Markman edited comment on DRILL-3189 at 9/15/15 6:25 PM:
--
Since we decided not to support "ROWS" syntax for now, it's not possible to
verify this fix.
Will revisit if we decide to support it.
was (Author: vicky):
Since we decided not to support "ROWS" syntax for now, it's not possible to
verify this fix.
> Disable ALLOW PARTIAL/DISALLOW PARTIAL in window function grammar
> -
>
> Key: DRILL-3189
> URL: https://issues.apache.org/jira/browse/DRILL-3189
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> It does not seem to be implemented on the drill side. Looks like Calcite
> specific grammar. Don't see it SQL Standard.
> Looks like wrong result:
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following disallow partial) from t2 order by
> a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | null|
> | 1 | null|
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | null|
> | 4 | null|
> | 5 | null|
> | 6 | null|
> | 7 | 14 |
> | 7 | 14 |
> | 8 | null|
> | 9 | null|
> +-+-+
> 13 rows selected (0.213 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following allow partial) from t2 order by a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | 0 |
> | 1 | 1 |
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | 3 |
> | 4 | 4 |
> | 5 | 5 |
> | 6 | 6 |
> | 7 | 14 |
> | 7 | 14 |
> | 8 | 8 |
> | 9 | 9 |
> +-+-+
> 13 rows selected (0.208 seconds)
> {code}
> {code}
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> disallow partial) from t2 order by a2;
> Error: PARSE ERROR: From line 1, column 53 to line 1, column 68: Cannot use
> DISALLOW PARTIAL with window based on RANGE
> [Error Id: 984c4b81-9eb0-401d-b36a-9580640b4a78 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3580) wrong plan for window function queries containing function(col1 + colb)
[
https://issues.apache.org/jira/browse/DRILL-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745973#comment-14745973
]
Victoria Markman commented on DRILL-3580:
-
Verified fixed in 1.2.0
#Fri Sep 11 05:38:24 UTC 2015
git.commit.id.abbrev=b525692
Tests added under: Functional/Passing/window_functions/multiple_partitions
> wrong plan for window function queries containing function(col1 + colb)
> ---
>
> Key: DRILL-3580
> URL: https://issues.apache.org/jira/browse/DRILL-3580
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Deneche A. Hakim
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> The following query has a wrong plan:
> {noformat}
> explain plan for select position_id, salary, sum(salary) over (partition by
> position_id), sum(position_id + salary) over (partition by position_id) from
> cp.`employee.json` limit 20;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 ProjectAllowDup(position_id=[$0], salary=[$1], EXPR$2=[$2],
> EXPR$3=[$3])
> 00-02SelectionVectorRemover
> 00-03 Limit(fetch=[20])
> 00-04Project(position_id=[$0], salary=[$1], w0$o0=[$2],
> w0$o00=[$4])
> 00-05 Window(window#0=[window(partition {0} order by [] range
> between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [SUM($3)])])
> 00-06Project(position_id=[$1], salary=[$2], w0$o0=[$3],
> $3=[+($1, $2)])
> 00-07 Window(window#0=[window(partition {1} order by []
> range between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [SUM($2)])])
> 00-08SelectionVectorRemover
> 00-09 Sort(sort0=[$1], dir0=[ASC])
> 00-10Project(T13¦¦*=[$0], position_id=[$1],
> salary=[$2])
> 00-11 Scan(groupscan=[EasyGroupScan
> [selectionRoot=classpath:/employee.json, numFiles=1, columns=[`*`],
> files=[classpath:/employee.json]]])
> {noformat}
> The plan contains 2 window operators which shouldn't be possible according to
> DRILL-3196.
> The results are also incorrect.
> Depending on which aggregation or window function used we get wrong results
> or an IndexOutOfBounds exception
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3580) wrong plan for window function queries containing function(col1 + colb)
[
https://issues.apache.org/jira/browse/DRILL-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3580.
---
> wrong plan for window function queries containing function(col1 + colb)
> ---
>
> Key: DRILL-3580
> URL: https://issues.apache.org/jira/browse/DRILL-3580
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Deneche A. Hakim
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> The following query has a wrong plan:
> {noformat}
> explain plan for select position_id, salary, sum(salary) over (partition by
> position_id), sum(position_id + salary) over (partition by position_id) from
> cp.`employee.json` limit 20;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 ProjectAllowDup(position_id=[$0], salary=[$1], EXPR$2=[$2],
> EXPR$3=[$3])
> 00-02SelectionVectorRemover
> 00-03 Limit(fetch=[20])
> 00-04Project(position_id=[$0], salary=[$1], w0$o0=[$2],
> w0$o00=[$4])
> 00-05 Window(window#0=[window(partition {0} order by [] range
> between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [SUM($3)])])
> 00-06Project(position_id=[$1], salary=[$2], w0$o0=[$3],
> $3=[+($1, $2)])
> 00-07 Window(window#0=[window(partition {1} order by []
> range between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [SUM($2)])])
> 00-08SelectionVectorRemover
> 00-09 Sort(sort0=[$1], dir0=[ASC])
> 00-10Project(T13¦¦*=[$0], position_id=[$1],
> salary=[$2])
> 00-11 Scan(groupscan=[EasyGroupScan
> [selectionRoot=classpath:/employee.json, numFiles=1, columns=[`*`],
> files=[classpath:/employee.json]]])
> {noformat}
> The plan contains 2 window operators which shouldn't be possible according to
> DRILL-3196.
> The results are also incorrect.
> Depending on which aggregation or window function used we get wrong results
> or an IndexOutOfBounds exception
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2949) TPC-DS queries 1 and 30 fail with CannotPlanException
[
https://issues.apache.org/jira/browse/DRILL-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2949:
Assignee: Abhishek Girish (was: Aman Sinha)
> TPC-DS queries 1 and 30 fail with CannotPlanException
> -
>
> Key: DRILL-2949
> URL: https://issues.apache.org/jira/browse/DRILL-2949
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Abhishek Girish
>Assignee: Abhishek Girish
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2949-Fix-CannotPlanException-for-TPC-DS-1-and-.patch
>
>
> TPC-DS queries 1 & 30 fail on recent master [Regression]:
> {code}
> SYSTEM ERROR: This query cannot be planned possibly due to either a cartesian
> join or an inequality join
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2949) TPC-DS queries 1 and 30 fail with CannotPlanException
[
https://issues.apache.org/jira/browse/DRILL-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745977#comment-14745977
]
Victoria Markman commented on DRILL-2949:
-
[~agirish],
I think you've already re-enabled these. Can you please confirm ?
Thanks,
Vicky.
> TPC-DS queries 1 and 30 fail with CannotPlanException
> -
>
> Key: DRILL-2949
> URL: https://issues.apache.org/jira/browse/DRILL-2949
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Abhishek Girish
>Assignee: Abhishek Girish
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2949-Fix-CannotPlanException-for-TPC-DS-1-and-.patch
>
>
> TPC-DS queries 1 & 30 fail on recent master [Regression]:
> {code}
> SYSTEM ERROR: This query cannot be planned possibly due to either a cartesian
> join or an inequality join
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2313) Query fails when one of the operands is a DATE literal without an explicit cast
[
https://issues.apache.org/jira/browse/DRILL-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746017#comment-14746017
]
Victoria Markman commented on DRILL-2313:
-
[~seanhychu],
(1) Is this expected behavior ?
{code}
0: jdbc:drill:schema=dfs> select * from t1 where c1 between cast('2015-01-01'
as date) and '2015-01-03 xxx';
+-++-+
| a1 | b1 | c1 |
+-++-+
| 1 | a | 2015-01-01 |
| 2 | b | 2015-01-02 |
| 3 | c | 2015-01-03 |
+-++-+
3 rows selected (0.222 seconds)
{code}
Postgres, for example, only tolerates trailing spaces:
{code}
postgres=# select * from t1 where c1 between cast('2015-01-01' as date) and
'2015-01-03 xx';
'ERROR: invalid input syntax for type date: "2015-01-03 xx"
LINE 1: ...1 where c1 between cast('2015-01-01' as date) and '2015-01-0...
^
postgres=# select * from t1 where c1 between cast('2015-01-01' as date) and
'2015-01-03';
a1 | b1 | c1
+---+
1 | a | 2015-01-01
2 | b | 2015-01-02
3 | c | 2015-01-03
(3 rows)
{code}
(2) In this fix, did we only implement implicit cast from string to date or
string to numeric should work as well ?
(3) It feels that we maybe need to document this behavior as well. What do you
think ?
> Query fails when one of the operands is a DATE literal without an explicit
> cast
> ---
>
> Key: DRILL-2313
> URL: https://issues.apache.org/jira/browse/DRILL-2313
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.8.0
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> For operations involving the date datatype, when one of the operands is a
> DATE literal without a cast, query fails.
> *The following query fails to validate:*
> {code:sql}
> SELECT
> *
> FROM
> date_dim
>
> WHEREd_date BETWEEN '2002-3-01' AND cast('2002-3-01' AS DATE)
> LIMIT 1;
> {code}
> Query failed: SqlValidatorException: Cannot apply 'BETWEEN' to arguments of
> type ' BETWEEN AND '. Supported form(s):
> ' BETWEEN AND '
> *The following query executes fine:*
> {code:sql}
> SELECT
> *
> FROM
> date_dim
>
> WHEREd_date BETWEEN '2002-3-01' AND
> '2002-3-01'
> LIMIT 1;
> {code}
> Both the queries execute fine on postgres
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2313) Query fails when one of the operands is a DATE literal without an explicit cast
[
https://issues.apache.org/jira/browse/DRILL-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746026#comment-14746026
]
Victoria Markman commented on DRILL-2313:
-
I answered question #2 myself:
{code}
0: jdbc:drill:schema=dfs> select * from t1 where a1 between 1 and cast('11'
as int);
+-++-+
| a1 | b1 | c1 |
+-++-+
| 1 | a | 2015-01-01 |
| 2 | b | 2015-01-02 |
| 3 | c | 2015-01-03 |
| 4 | null | 2015-01-04 |
| 5 | e | 2015-01-05 |
| 6 | f | 2015-01-06 |
| 7 | g | 2015-01-07 |
| 9 | i | null|
| 10 | j | 2015-01-10 |
+-++-+
9 rows selected (0.215 seconds)
{code}
> Query fails when one of the operands is a DATE literal without an explicit
> cast
> ---
>
> Key: DRILL-2313
> URL: https://issues.apache.org/jira/browse/DRILL-2313
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.8.0
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> For operations involving the date datatype, when one of the operands is a
> DATE literal without a cast, query fails.
> *The following query fails to validate:*
> {code:sql}
> SELECT
> *
> FROM
> date_dim
>
> WHEREd_date BETWEEN '2002-3-01' AND cast('2002-3-01' AS DATE)
> LIMIT 1;
> {code}
> Query failed: SqlValidatorException: Cannot apply 'BETWEEN' to arguments of
> type ' BETWEEN AND '. Supported form(s):
> ' BETWEEN AND '
> *The following query executes fine:*
> {code:sql}
> SELECT
> *
> FROM
> date_dim
>
> WHEREd_date BETWEEN '2002-3-01' AND
> '2002-3-01'
> LIMIT 1;
> {code}
> Both the queries execute fine on postgres
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (DRILL-2332) Drill should be consistent with Implicit casting rules across data formats
[
https://issues.apache.org/jira/browse/DRILL-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-2332:
-
Sean,
Fix for DRILL-2313 I believe, makes this case work now:
t1 is parquet file:
message root {
optional int32 a1;
optional binary b1 (UTF8);
optional int32 c1 (DATE);
}
{code}
0: jdbc:drill:schema=dfs> select * from t1 where c1 in ('2015-01-01');
+-++-+
| a1 | b1 | c1 |
+-++-+
| 1 | a | 2015-01-01 |
+-++-+
1 row selected (0.191 seconds)
0: jdbc:drill:schema=dfs> create view v1(a1, b1, c1) as select cast(a1 as int),
cast(b1 as varchar(20)), cast(c1 as date) from t1;
+---++
| ok | summary |
+---++
| true | View 'v1' created successfully in 'dfs.subqueries' schema |
+---++
1 row selected (0.226 seconds)
0: jdbc:drill:schema=dfs> describe v1;
+--++--+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+--++--+
| a1 | INTEGER| YES |
| b1 | CHARACTER VARYING | YES |
| c1 | DATE | YES |
+--++--+
3 rows selected (0.334 seconds)
0: jdbc:drill:schema=dfs> select * from v1 where c1 in ('2015-01-01');
+-++-+
| a1 | b1 | c1 |
+-++-+
| 1 | a | 2015-01-01 |
+-++-+
1 row selected (0.241 seconds)
{code}
We should resolve it as a duplicate of DRILL-2313 (if you agree) and
extensively test implicit cast, This is a pretty significant change in drill's
behavior.
> Drill should be consistent with Implicit casting rules across data formats
> --
>
> Key: DRILL-2332
> URL: https://issues.apache.org/jira/browse/DRILL-2332
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Currently, the outcome of a query with a filter on a column comparing it with
> a literal, depends on the underlying data format.
> *Parquet*
> {code:sql}
> select * from date_dim where d_month_seq ='1193' limit 1;
> [Succeeds]
> select * from date_dim where d_date in ('1999-06-30') limit 1;
> [Succeeds]
> {code}
> *View on top of text:*
> {code:sql}
> select * from date_dim where d_date in ('1999-06-30') limit 1;
> Query failed: SqlValidatorException: Values passed to IN operator must have
> compatible types
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> select * from date_dim where d_month_seq ='1193' limit 1;
> Query failed: SqlValidatorException: Cannot apply '=' to arguments of type
> ' = '. Supported form(s): ' =
> '
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> I understand that in the case of View on Text, SQL validation fails at the
> Optiq layer.
> But from the perspective of an end-user, Drill's behavior must be consistent
> across data formats. Also having a view by definition should abstract out
> this information.
> Here, both the view and parquet were created with type information.
> *Parquet-meta*
> {code}
> parquet-schema /mapr/abhi311/data/parquet/tpcds/scale1/date_dim/0_0_0.parquet
> message root {
> optional int32 d_date_sk;
> optional binary d_date_id (UTF8);
> optional binary d_date (UTF8);
> optional int32 d_month_seq;
> optional int32 d_week_seq;
> optional int32 d_quarter_seq;
> optional int32 d_year;
> optional int32 d_dow;
> optional int32 d_moy;
> optional int32 d_dom;
> optional int32 d_qoy;
> optional int32 d_fy_year;
> optional int32 d_fy_quarter_seq;
> optional int32 s_fy_week_seq;
> optional binary d_day_name (UTF8);
> optional binary d_quarter_name (UTF8);
> optional binary d_holiday (UTF8);
> optional binary d_weekend (UTF8);
> optional binary d_following_holiday (UTF8);
> optional int32 d_first_dom;
> optional int32 d_last_dom;
> optional int32 d_same_day_ly;
> optional int32 d_same_day_lq;
> optional binary d_current_day (UTF8);
> optional binary d_current_week (UTF8);
> optional binary d_current_month (UTF8);
> optional binary d_current_quarter (UTF8);
> optional binary d_current_year (UTF8);
> }
> {code}
> *Describe View*
> {code:sql}
> > describe date_dim;
> +-++-+
> | COLUM
[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3238:
Fix Version/s: (was: Future)
1.2.0
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746596#comment-14746596
]
Victoria Markman commented on DRILL-2748:
-
[~jni]
Costing theory may not be correct: tried with tpcds sf100, filter is not pushed
down :(
{code}
0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select
ss_quantity, ss_store_sk, avg(ss_quantity) from store sales group by
ss_quantity, ss_store_sk) as sq(x, y, z) where x = 10;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(x=[$0], y=[$1], z=[$2])
00-02Project(x=[$0], y=[$1], z=[$2])
00-03 Project(ss_quantity=[$0], ss_store_sk=[$1],
EXPR$2=[CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY NOT NULL])
00-04SelectionVectorRemover
00-05 Filter(condition=[=($0, 10)])
00-06HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
agg#1=[COUNT($0)])
00-07 Project(ss_quantity=[$1], ss_store_sk=[$0])
00-08Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tpcds100/parquet/store]],
selectionRoot=maprfs:/tpcds100/parquet/store, numFiles=1,
columns=[`ss_quantity`, `ss_store_sk`]]])
{code}
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746599#comment-14746599
]
Victoria Markman commented on DRILL-2748:
-
[~jni]
I tried with tpcds sf100, filter is not pushed down as well as in the original
query:
{code}
0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select
ss_quantity, ss_store_sk, avg(ss_quantity) from store sales group by
ss_quantity, ss_store_sk) as sq(x, y, z) where x = 10;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(x=[$0], y=[$1], z=[$2])
00-02Project(x=[$0], y=[$1], z=[$2])
00-03 Project(ss_quantity=[$0], ss_store_sk=[$1],
EXPR$2=[CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY NOT NULL])
00-04SelectionVectorRemover
00-05 Filter(condition=[=($0, 10)])
00-06HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
agg#1=[COUNT($0)])
00-07 Project(ss_quantity=[$1], ss_store_sk=[$0])
00-08Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tpcds100/parquet/store]],
selectionRoot=maprfs:/tpcds100/parquet/store, numFiles=1,
columns=[`ss_quantity`, `ss_store_sk`]]])
{code}
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-2748:
-
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2748:
Comment: was deleted
(was: [~jni]
I tried with tpcds sf100, filter is not pushed down as well as in the original
query:
{code}
0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select
ss_quantity, ss_store_sk, avg(ss_quantity) from store sales group by
ss_quantity, ss_store_sk) as sq(x, y, z) where x = 10;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(x=[$0], y=[$1], z=[$2])
00-02Project(x=[$0], y=[$1], z=[$2])
00-03 Project(ss_quantity=[$0], ss_store_sk=[$1],
EXPR$2=[CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY NOT NULL])
00-04SelectionVectorRemover
00-05 Filter(condition=[=($0, 10)])
00-06HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
agg#1=[COUNT($0)])
00-07 Project(ss_quantity=[$1], ss_store_sk=[$0])
00-08Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///tpcds100/parquet/store]],
selectionRoot=maprfs:/tpcds100/parquet/store, numFiles=1,
columns=[`ss_quantity`, `ss_store_sk`]]])
{code})
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Add-optimizer-rule-to-push-filter-past-ag.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3783) Incorrect results : COUNT() over results returned by UNION ALL
[
https://issues.apache.org/jira/browse/DRILL-3783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746800#comment-14746800
]
Victoria Markman commented on DRILL-3783:
-
{code}
postgres=# select count(a1) from ((select a1 from t1 union all select a1 from
t1)) x;
count
---
18
(1 row)
{code}
Who is going to read the standard ? :)
> Incorrect results : COUNT() over results returned by UNION ALL
>
>
> Key: DRILL-3783
> URL: https://issues.apache.org/jira/browse/DRILL-3783
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.2.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Fix For: 1.2.0
>
>
> Count over results returned union all query, returns incorrect results. The
> below query returned an Exception (please se DRILL-2637) that JIRA was marked
> as fixed, however the query returns incorrect results.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select count(c1) from (select cast(columns[0]
> as int) c1 from `testWindow.csv`) union all (select cast(columns[0] as int)
> c2 from `testWindow.csv`);
> +-+
> | EXPR$0 |
> +-+
> | 11 |
> | 100 |
> | 10 |
> | 2 |
> | 50 |
> | 55 |
> | 67 |
> | 113 |
> | 119 |
> | 89 |
> | 57 |
> | 61 |
> +-+
> 12 rows selected (0.753 seconds)
> {code}
> Results returned by the query on LHS and RHS of Union all operator are
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select cast(columns[0] as int) c1 from
> `testWindow.csv`;
> +--+
> | c1 |
> +--+
> | 100 |
> | 10 |
> | 2|
> | 50 |
> | 55 |
> | 67 |
> | 113 |
> | 119 |
> | 89 |
> | 57 |
> | 61 |
> +--+
> 11 rows selected (0.197 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select cast(columns[0] as int) c2 from
> `testWindow.csv`;
> +--+
> | c2 |
> +--+
> | 100 |
> | 10 |
> | 2|
> | 50 |
> | 55 |
> | 67 |
> | 113 |
> | 119 |
> | 89 |
> | 57 |
> | 61 |
> +--+
> 11 rows selected (0.173 seconds)
> {code}
> Note that enclosing the queries within correct parentheses returns correct
> results. We do not want to return incorrect results to user when the
> parentheses are missing.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select count(c1) from ((select cast(columns[0]
> as int) c1 from `testWindow.csv`) union all (select cast(columns[0] as int)
> c2 from `testWindow.csv`));
> +-+
> | EXPR$0 |
> +-+
> | 22 |
> +-+
> 1 row selected (0.234 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791093#comment-14791093
]
Victoria Markman commented on DRILL-3238:
-
Verified fixed in 1.2.0
#Tue Sep 15 22:16:23 UTC 2015
git.commit.id.abbrev=48bc0b9
Test added under: Functional/Passing/window_functions/bugs
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3238.
---
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3679) IOB Exception : when window functions used in outer and inner query
[
https://issues.apache.org/jira/browse/DRILL-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3679.
---
> IOB Exception : when window functions used in outer and inner query
> ---
>
> Key: DRILL-3679
> URL: https://issues.apache.org/jira/browse/DRILL-3679
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
> Environment: private-branch
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Jinfeng Ni
> Labels: window_function
> Fix For: 1.2.0
>
>
> IOB Exception seen when two different window functions are used in inner and
> outer queries.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rnum, position_id, ntile(4) over(order
> by position_id) from (select position_id, row_number() over(order by
> position_id) as rnum from cp.`employee.json`);
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> IndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))
> Fragment 0:0
> [Error Id: 8e0cbf82-842d-4fa7-ab0d-1d982a3d6c24 on centos-03.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3679) IOB Exception : when window functions used in outer and inner query
[
https://issues.apache.org/jira/browse/DRILL-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791355#comment-14791355
]
Victoria Markman commented on DRILL-3679:
-
Verified fixed in 1.2.0
#Wed Sep 16 00:30:47 UTC 2015
git.commit.id.abbrev=9afcf61
Test added to: Functional/Passing/window_functions/bugs
> IOB Exception : when window functions used in outer and inner query
> ---
>
> Key: DRILL-3679
> URL: https://issues.apache.org/jira/browse/DRILL-3679
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.2.0
> Environment: private-branch
> https://github.com/adeneche/incubator-drill/tree/new-window-funcs
>Reporter: Khurram Faraaz
>Assignee: Jinfeng Ni
> Labels: window_function
> Fix For: 1.2.0
>
>
> IOB Exception seen when two different window functions are used in inner and
> outer queries.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select rnum, position_id, ntile(4) over(order
> by position_id) from (select position_id, row_number() over(order by
> position_id) as rnum from cp.`employee.json`);
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> IndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))
> Fragment 0:0
> [Error Id: 8e0cbf82-842d-4fa7-ab0d-1d982a3d6c24 on centos-03.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-3238:
-
Sean,
I just realized that test failed for me and query now returns wrong result.
Please see below:
-- wrong result caused by a wrong plan (notice two window operators)
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > a2,
. . . . . . . . . . . . > sum(a2) over (w),
. . . . . . . . . . . . > count(*) over w
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2
. . . . . . . . . . . . > window w as (partition by a2 order by a2);
+-+-+-+
| a2 | EXPR$1 | EXPR$2 |
+-+-+-+
| 0 | 0 | 1 |
| 1 | 1 | 1 |
| 2 | 7 | 3 |
| 2 | 7 | 3 |
| 2 | 7 | 3 |
| 3 | 10 | 1 |
| 4 | 14 | 1 |
| 5 | 19 | 1 |
| 6 | 25 | 1 |
| 7 | 39 | 2 |
| 7 | 39 | 2 |
| 8 | 47 | 1 |
| 9 | 56 | 1 |
+-+-+-+
13 rows selected (0.305 seconds)
0: jdbc:drill:schema=dfs> explain plan for select a2, sum(a2) over(w), count(*)
over w from t2 window w as (partition by a2 order by a2);
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(a2=[$0], EXPR$1=[$1], EXPR$2=[$2])
00-02Window(window#0=[window(partition {0} order by [0] range between
UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT()])])
00-03 SelectionVectorRemover
00-04Sort(sort0=[$0], sort1=[$0], dir0=[ASC], dir1=[ASC])
00-05 Window(window#0=[window(partition {} order by [0] range
between UNBOUNDED PRECEDING and CURRENT ROW aggs [SUM($0)])])
00-06SelectionVectorRemover
00-07 Sort(sort0=[$0], dir0=[ASC])
00-08Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/t2]],
selectionRoot=maprfs:/drill/testdata/subqueries/t2, numFiles=1,
columns=[`a2`]]])
{code}
-- correct
{code}
0: jdbc:drill:schema=dfs> select a2, sum(a2) over (partition by a2 order by a2)
from t2;
+-+-+
| a2 | EXPR$1 |
+-+-+
| 0 | 0 |
| 1 | 1 |
| 2 | 6 |
| 2 | 6 |
| 2 | 6 |
| 3 | 3 |
| 4 | 4 |
| 5 | 5 |
| 6 | 6 |
| 7 | 14 |
| 7 | 14 |
| 8 | 8 |
| 9 | 9 |
+-+-+
13 rows selected (0.303 seconds)
{code}
--correct
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > a2,
. . . . . . . . . . . . > sum(a2) over (partition by a2 order by a2),
. . . . . . . . . . . . > count(*) over(partition by a2 order by a2)
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2;
+-+-+-+
| a2 | EXPR$1 | EXPR$2 |
+-+-+-+
| 0 | 0 | 1 |
| 1 | 1 | 1 |
| 2 | 6 | 3 |
| 2 | 6 | 3 |
| 2 | 6 | 3 |
| 3 | 3 | 1 |
| 4 | 4 | 1 |
| 5 | 5 | 1 |
| 6 | 6 | 1 |
| 7 | 14 | 2 |
| 7 | 14 | 2 |
| 8 | 8 | 1 |
| 9 | 9 | 1 |
+-+-+-+
13 rows selected (0.324 seconds)
{code}
-- correct
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > a2,
. . . . . . . . . . . . > sum(a2) over w,
. . . . . . . . . . . . > count(*) over w
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2
. . . . . . . . . . . . > window w as (partition by a2 order by a2);
+-+-+-+
| a2 | EXPR$1 | EXPR$2 |
+-+-+-+
| 0 | 0 | 1 |
| 1 | 1 | 1 |
| 2 | 6 | 3 |
| 2 | 6 | 3 |
| 2 | 6 | 3 |
| 3 | 3 | 1 |
| 4 | 4 | 1 |
| 5 | 5 | 1 |
| 6 | 6 | 1 |
| 7 | 14 | 2 |
| 7 | 14 | 2 |
| 8 | 8 | 1 |
| 9 | 9 | 1 |
+-+-+-+
13 rows selected (0.29 seconds)
{code}
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order b
[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3238:
Priority: Critical (was: Major)
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803374#comment-14803374
]
Victoria Markman commented on DRILL-3238:
-
I only marked this case as critical, because it is wrong result. However, all
the straightforward cases work, I don't think it is that imprortant.
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804671#comment-14804671
]
Victoria Markman commented on DRILL-3238:
-
Lowering priority and moving to 1.3.0 - rare grammar.
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3238:
Priority: Major (was: Critical)
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.2.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3238:
Fix Version/s: (was: 1.2.0)
1.3.0
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.3.0
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3156) Calcite tracing is broken in Drill
[
https://issues.apache.org/jira/browse/DRILL-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875873#comment-14875873
]
Victoria Markman commented on DRILL-3156:
-
This requires unit test in calcite.
> Calcite tracing is broken in Drill
> ---
>
> Key: DRILL-3156
> URL: https://issues.apache.org/jira/browse/DRILL-3156
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3156-Fix-Calcite-trace-by-registering-RelMetad.patch
>
>
> Apache Calcite's tracing functionality plays an important role when we debug
> issues related to either logical planning or physical planning in Drill.
> However, since Drill-705 introduced a HepPlanner to do query planning for
> Window Function, the tracing functionality is broken.
> For instance, if I turn on Calcite tracing and set the tracing level to
> "FINER", as following:
> {code}
> org.apache.calcite.plan.RelOptPlanner.level = FINER
> {code}
> When submit a query to Drill, Drill hit AssertionError.
> {code}
> SYSTEM ERROR: java.lang.AssertionError: Internal error: should never get here
> [Error Id: 3410c3b4-726f-487b-a1c4-287b6ae724aa on 172.16.0.120:31010]
> {code}
> The root cause of this Assertion is here :
> {code}
> Caused by: java.lang.AssertionError: Internal error: should never get here
> at org.apache.calcite.util.Util.newInternal(Util.java:775)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.plan.hep.HepRelVertex.computeSelfCost(HepRelVertex.java:68)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost(RelMdPercentageOriginalRows.java:165)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3156) Calcite tracing is broken in Drill
[
https://issues.apache.org/jira/browse/DRILL-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3156.
---
> Calcite tracing is broken in Drill
> ---
>
> Key: DRILL-3156
> URL: https://issues.apache.org/jira/browse/DRILL-3156
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3156-Fix-Calcite-trace-by-registering-RelMetad.patch
>
>
> Apache Calcite's tracing functionality plays an important role when we debug
> issues related to either logical planning or physical planning in Drill.
> However, since Drill-705 introduced a HepPlanner to do query planning for
> Window Function, the tracing functionality is broken.
> For instance, if I turn on Calcite tracing and set the tracing level to
> "FINER", as following:
> {code}
> org.apache.calcite.plan.RelOptPlanner.level = FINER
> {code}
> When submit a query to Drill, Drill hit AssertionError.
> {code}
> SYSTEM ERROR: java.lang.AssertionError: Internal error: should never get here
> [Error Id: 3410c3b4-726f-487b-a1c4-287b6ae724aa on 172.16.0.120:31010]
> {code}
> The root cause of this Assertion is here :
> {code}
> Caused by: java.lang.AssertionError: Internal error: should never get here
> at org.apache.calcite.util.Util.newInternal(Util.java:775)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.plan.hep.HepRelVertex.computeSelfCost(HepRelVertex.java:68)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows.getNonCumulativeCost(RelMdPercentageOriginalRows.java:165)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3716) Drill should push filter past aggregate in order to improve query performance.
[
https://issues.apache.org/jira/browse/DRILL-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14875897#comment-14875897
]
Victoria Markman commented on DRILL-3716:
-
I've tested this with 1.2.0 and this particular case works now.
#Wed Sep 16 00:30:47 UTC 2015
git.commit.id.abbrev=9afcf61
Working on a regression test suite for filter pushdown.
{code}
0: jdbc:drill:schema=dfs> explain plan for select n_regionkey, cnt from
. . . . . . . . . . . . > (select n_regionkey, count(*) cnt
. . . . . . . . . . . . > from (select n.n_nationkey, n.n_regionkey,
n.n_name
. . . . . . . . . . . . >from cp.`tpch/nation.parquet` n
. . . . . . . . . . . . > left join
. . . . . . . . . . . . >cp.`tpch/region.parquet` r
. . . . . . . . . . . . > on n.n_regionkey = r.r_regionkey)
. . . . . . . . . . . . >group by n_regionkey)
. . . . . . . . . . . . > where n_regionkey = 2;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(n_regionkey=[$0], cnt=[$1])
00-02Project(n_regionkey=[$0], cnt=[$1])
00-03 StreamAgg(group=[{0}], cnt=[COUNT()])
00-04Sort(sort0=[$0], dir0=[ASC])
00-05 Project(n_regionkey=[$0])
00-06Project(n_regionkey=[$1], r_regionkey=[$0])
00-07 HashJoin(condition=[=($1, $0)], joinType=[right])
00-09Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/region.parquet]],
selectionRoot=classpath:/tpch/region.parquet, numFiles=1,
columns=[`r_regionkey`]]])
00-08SelectionVectorRemover
00-10 Filter(condition=[=($0, 2)])
00-11Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
selectionRoot=classpath:/tpch/nation.parquet, numFiles=1,
columns=[`n_regionkey`]]])
{code}
> Drill should push filter past aggregate in order to improve query performance.
> --
>
> Key: DRILL-3716
> URL: https://issues.apache.org/jira/browse/DRILL-3716
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
>
> For the following query which has a filter on top of an aggregation, Drill's
> currently push the filter pass through the aggregation. As a result, we may
> miss some optimization opportunity. For instance, such filter could
> potentially been pushed into scan if it qualifies for partition pruning.
> For the following query:
> {code}
> select n_regionkey, cnt from
> (select n_regionkey, count(*) cnt
> from (select n.n_nationkey, n.n_regionkey, n.n_name
>from cp.`tpch/nation.parquet` n
> left join
>cp.`tpch/region.parquet` r
> on n.n_regionkey = r.r_regionkey)
>group by n_regionkey)
> where n_regionkey = 2;
> {code}
> The current plan shows a filter (00-04) on top of aggregation(00-05). The
> better plan would have the filter pushed pass the aggregation.
> The root cause of this problem is Drill's ruleset does not include
> FilterAggregateTransoposeRule from Calcite library.
> {code}
> 00-01 Project(n_regionkey=[$0], cnt=[$1])
> 00-02Project(n_regionkey=[$0], cnt=[$1])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 2)])
> 00-05 StreamAgg(group=[{0}], cnt=[COUNT()])
> 00-06Project(n_regionkey=[$0])
> 00-07 MergeJoin(condition=[=($0, $1)], joinType=[left])
> 00-09SelectionVectorRemover
> 00-11 Sort(sort0=[$0], dir0=[ASC])
> 00-13Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
> selectionRoot=classpath:/tpch/nation.parquet, numFiles=1,
> columns=[`n_regionkey`]]])
> 00-08SelectionVectorRemover
> 00-10 Sort(sort0=[$0], dir0=[ASC])
> 00-12Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/region.parquet]],
> selectionRoot=classpath:/tpch/region.parquet, numFiles=1,
> columns=[`r_regionkey`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3716) Drill should push filter past aggregate in order to improve query performance.
[
https://issues.apache.org/jira/browse/DRILL-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3716.
---
> Drill should push filter past aggregate in order to improve query performance.
> --
>
> Key: DRILL-3716
> URL: https://issues.apache.org/jira/browse/DRILL-3716
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
>
> For the following query which has a filter on top of an aggregation, Drill's
> currently push the filter pass through the aggregation. As a result, we may
> miss some optimization opportunity. For instance, such filter could
> potentially been pushed into scan if it qualifies for partition pruning.
> For the following query:
> {code}
> select n_regionkey, cnt from
> (select n_regionkey, count(*) cnt
> from (select n.n_nationkey, n.n_regionkey, n.n_name
>from cp.`tpch/nation.parquet` n
> left join
>cp.`tpch/region.parquet` r
> on n.n_regionkey = r.r_regionkey)
>group by n_regionkey)
> where n_regionkey = 2;
> {code}
> The current plan shows a filter (00-04) on top of aggregation(00-05). The
> better plan would have the filter pushed pass the aggregation.
> The root cause of this problem is Drill's ruleset does not include
> FilterAggregateTransoposeRule from Calcite library.
> {code}
> 00-01 Project(n_regionkey=[$0], cnt=[$1])
> 00-02Project(n_regionkey=[$0], cnt=[$1])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 2)])
> 00-05 StreamAgg(group=[{0}], cnt=[COUNT()])
> 00-06Project(n_regionkey=[$0])
> 00-07 MergeJoin(condition=[=($0, $1)], joinType=[left])
> 00-09SelectionVectorRemover
> 00-11 Sort(sort0=[$0], dir0=[ASC])
> 00-13Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/nation.parquet]],
> selectionRoot=classpath:/tpch/nation.parquet, numFiles=1,
> columns=[`n_regionkey`]]])
> 00-08SelectionVectorRemover
> 00-10 Sort(sort0=[$0], dir0=[ASC])
> 00-12Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=classpath:/tpch/region.parquet]],
> selectionRoot=classpath:/tpch/region.parquet, numFiles=1,
> columns=[`r_regionkey`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3801) Extend support for scalar subqueries
Victoria Markman created DRILL-3801:
---
Summary: Extend support for scalar subqueries
Key: DRILL-3801
URL: https://issues.apache.org/jira/browse/DRILL-3801
Project: Apache Drill
Issue Type: New Feature
Reporter: Victoria Markman
In current drill implementation we only consider subquery scalar when scalar
aggregate is present and as I discovered recently, we also consider subquery to
return scalar result if you have 1 column, and 1 row (limit 1)
However, if subquery does return one row during runtime, we currently don't
detect that and we throw an error early during planning time:
{code}
0: jdbc:drill:schema=dfs> select * from t2 where a2 = 0;
+-+--+-+
| a2 | b2 | c2 |
+-+--+-+
| 0 | zzz | 2014-12-31 |
+-+--+-+
1 row selected (0.283 seconds)
0: jdbc:drill:schema=dfs> select * from t1 where a1 >= (select a2 from t2 where
a2 = 0);
Error: UNSUPPORTED_OPERATION ERROR: Non-scalar sub-query used in an expression
See Apache Drill JIRA: DRILL-1937
[Error Id: 9e26af25-b0c4-42f9-a839-b76f759c94ab on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
Because of this limitation we can't run TPCDS query 44 (inner subquery has
just one group):
{code}
...
...
SELECT ss_item_sk item_sk,
Avg(ss_net_profit) rank_col
FROM store_sales ss1
WHERE ss_store_sk = 4
GROUP BY ss_item_sk
HAVING Avg(ss_net_profit) > 0.9 *
(SELECT Avg(ss_net_profit)
rank_col
FROM store_sales
WHERE ss_store_sk = 4
AND ss_cdemo_sk IS
NULL
GROUP BY ss_store_sk))V2)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3802) Throw unsupported error for ROLLUP/GROUPING
Victoria Markman created DRILL-3802:
---
Summary: Throw unsupported error for ROLLUP/GROUPING
Key: DRILL-3802
URL: https://issues.apache.org/jira/browse/DRILL-3802
Project: Apache Drill
Issue Type: New Feature
Reporter: Victoria Markman
I believe that this is the cause of assertions in TPCDS #36, #67
{code}
SELECT Sum(ss_net_profit) / Sum(ss_ext_sales_price) AS
gross_margin,
i_category,
i_class,
Grouping(i_category) + Grouping(i_class) AS
lochierarchy,
Rank()
OVER (
partition BY Grouping(i_category)+Grouping(i_class), CASE
WHEN Grouping(
i_class) = 0 THEN i_category END
ORDER BY Sum(ss_net_profit)/Sum(ss_ext_sales_price) ASC) AS
rank_within_parent
FROM store_sales,
date_dim d1,
item,
store
WHERE d1.d_year = 2000
AND d1.d_date_sk = ss_sold_date_sk
AND i_item_sk = ss_item_sk
AND s_store_sk = ss_store_sk
AND s_state IN ( 'TN', 'TN', 'TN', 'TN',
'TN', 'TN', 'TN', 'TN' )
GROUP BY rollup( i_category, i_class )
ORDER BY lochierarchy DESC,
CASE
WHEN lochierarchy = 0 THEN i_category
END,
rank_within_parent
LIMIT 100;
Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated:
conversion result not null
[Error Id: 6afae7ce-c426-44f3-a600-aa34ab7632a1 on ucs-node5.perf.lab:31010]
(state=,code=0)
java.sql.SQLException: SYSTEM ERROR: AssertionError: Internal error: invariant
violated: conversion result not null
[Error Id: 6afae7ce-c426-44f3-a600-aa34ab7632a1 on ucs-node5.perf.lab:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
at
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1359)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:74)
at
net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
at
net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
at
net.hydromatic.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:338)
at net.hydromatic.avatica.AvaticaStatement.execute(AvaticaStatement.java:69)
at
org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:86)
at sqlline.Commands.execute(Commands.java:841)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:738)
at sqlline.SqlLine.runCommands(SqlLine.java:1641)
at sqlline.Commands.run(Commands.java:1304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
at sqlline.SqlLine.dispatch(SqlLine.java:734)
at sqlline.SqlLine.initArgs(SqlLine.java:544)
at sqlline.SqlLine.begin(SqlLine.java:587)
at sqlline.SqlLine.start(SqlLine.java:366)
at sqlline.SqlLine.main(SqlLine.java:259)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
ERROR: AssertionError: Internal error: invariant violated: conversion result
not null
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2398) IS NOT DISTINCT FROM predicate returns incorrect result when used as a join filter
[
https://issues.apache.org/jira/browse/DRILL-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876025#comment-14876025
]
Victoria Markman commented on DRILL-2398:
-
I verified that this query returns correct result with 1.2.0
#Wed Sep 16 00:30:47 UTC 2015
git.commit.id.abbrev=9afcf61
The problem here is not "IS NOT DISTINCT" predicate, but count(*) over empty
result set that should return 0 and not NULL.
This is a duplicate of https://issues.apache.org/jira/browse/DRILL-2277
> IS NOT DISTINCT FROM predicate returns incorrect result when used as a join
> filter
> --
>
> Key: DRILL-2398
> URL: https://issues.apache.org/jira/browse/DRILL-2398
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: j1.parquet, j2.parquet
>
>
> count(*) should return 0 and not NULL
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > j1 INNER JOIN j2 ON
> . . . . . . . . . . . . > ( j1.c_double = j2.c_double)
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > j1.c_bigint IS NOT DISTINCT FROM j2.c_bigint
> . . . . . . . . . . . . > ;
> ++
> | EXPR$0 |
> ++
> ++
> {code}
> These are the values in the table
> {code}
> 0: jdbc:drill:schema=dfs> select j1.c_bigint, j2.c_bigint, count(*) from j1
> INNER JOIN j2 ON (j1.c_double = j2.c_double) group by j1.c_bigint,
> j2.c_bigint;
> ++++
> | c_bigint | c_bigint1 | EXPR$1 |
> ++++
> | 460194667 | -498749284 | 1 |
> | 464547172 | -498828740 | 1 |
> | 467451850 | -498966611 | 2 |
> | 471050029 | -499154096 | 3 |
> | 472873799 | -499233550 | 3 |
> | 475698977 | -499395929 | 2 |
> | 478986584 | -499564607 | 1 |
> | 488139464 | -499763274 | 3 |
> | 498214699 | -499871720 | 2 |
> ++++
> 9 rows selected (0.339 seconds)
> {code}
> IS DISTINCT FROM predicate returns correct result
> {code}
> select
> count(*)
> from
> j1 INNER JOIN j2 ON
> ( j1.c_double = j2.c_double)
> where
> j1.c_bigint IS DISTINCT FROM j2.c_bigint
> {code}
> Explain plan for query that returns incorrect result:
> {code}
> 00-01 StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-02Project($f0=[0])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[CAST(CASE(IS NULL($1), IS NULL($3), IS
> NULL($3), IS NULL($1), =($1, $3))):BOOLEAN NOT NULL])
> 00-05 HashJoin(condition=[=($0, $2)], joinType=[inner])
> 00-07Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j1]],
> selectionRoot=/joins/j1, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> 00-06Project(c_double0=[$0], c_bigint0=[$1])
> 00-08 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j2]],
> selectionRoot=/joins/j2, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-2398) IS NOT DISTINCT FROM predicate returns incorrect result when used as a join filter
[
https://issues.apache.org/jira/browse/DRILL-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman resolved DRILL-2398.
-
Resolution: Duplicate
> IS NOT DISTINCT FROM predicate returns incorrect result when used as a join
> filter
> --
>
> Key: DRILL-2398
> URL: https://issues.apache.org/jira/browse/DRILL-2398
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: j1.parquet, j2.parquet
>
>
> count(*) should return 0 and not NULL
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > j1 INNER JOIN j2 ON
> . . . . . . . . . . . . > ( j1.c_double = j2.c_double)
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > j1.c_bigint IS NOT DISTINCT FROM j2.c_bigint
> . . . . . . . . . . . . > ;
> ++
> | EXPR$0 |
> ++
> ++
> {code}
> These are the values in the table
> {code}
> 0: jdbc:drill:schema=dfs> select j1.c_bigint, j2.c_bigint, count(*) from j1
> INNER JOIN j2 ON (j1.c_double = j2.c_double) group by j1.c_bigint,
> j2.c_bigint;
> ++++
> | c_bigint | c_bigint1 | EXPR$1 |
> ++++
> | 460194667 | -498749284 | 1 |
> | 464547172 | -498828740 | 1 |
> | 467451850 | -498966611 | 2 |
> | 471050029 | -499154096 | 3 |
> | 472873799 | -499233550 | 3 |
> | 475698977 | -499395929 | 2 |
> | 478986584 | -499564607 | 1 |
> | 488139464 | -499763274 | 3 |
> | 498214699 | -499871720 | 2 |
> ++++
> 9 rows selected (0.339 seconds)
> {code}
> IS DISTINCT FROM predicate returns correct result
> {code}
> select
> count(*)
> from
> j1 INNER JOIN j2 ON
> ( j1.c_double = j2.c_double)
> where
> j1.c_bigint IS DISTINCT FROM j2.c_bigint
> {code}
> Explain plan for query that returns incorrect result:
> {code}
> 00-01 StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-02Project($f0=[0])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[CAST(CASE(IS NULL($1), IS NULL($3), IS
> NULL($3), IS NULL($1), =($1, $3))):BOOLEAN NOT NULL])
> 00-05 HashJoin(condition=[=($0, $2)], joinType=[inner])
> 00-07Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j1]],
> selectionRoot=/joins/j1, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> 00-06Project(c_double0=[$0], c_bigint0=[$1])
> 00-08 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j2]],
> selectionRoot=/joins/j2, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-2398) IS NOT DISTINCT FROM predicate returns incorrect result when used as a join filter
[
https://issues.apache.org/jira/browse/DRILL-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2398.
---
> IS NOT DISTINCT FROM predicate returns incorrect result when used as a join
> filter
> --
>
> Key: DRILL-2398
> URL: https://issues.apache.org/jira/browse/DRILL-2398
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: j1.parquet, j2.parquet
>
>
> count(*) should return 0 and not NULL
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > j1 INNER JOIN j2 ON
> . . . . . . . . . . . . > ( j1.c_double = j2.c_double)
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > j1.c_bigint IS NOT DISTINCT FROM j2.c_bigint
> . . . . . . . . . . . . > ;
> ++
> | EXPR$0 |
> ++
> ++
> {code}
> These are the values in the table
> {code}
> 0: jdbc:drill:schema=dfs> select j1.c_bigint, j2.c_bigint, count(*) from j1
> INNER JOIN j2 ON (j1.c_double = j2.c_double) group by j1.c_bigint,
> j2.c_bigint;
> ++++
> | c_bigint | c_bigint1 | EXPR$1 |
> ++++
> | 460194667 | -498749284 | 1 |
> | 464547172 | -498828740 | 1 |
> | 467451850 | -498966611 | 2 |
> | 471050029 | -499154096 | 3 |
> | 472873799 | -499233550 | 3 |
> | 475698977 | -499395929 | 2 |
> | 478986584 | -499564607 | 1 |
> | 488139464 | -499763274 | 3 |
> | 498214699 | -499871720 | 2 |
> ++++
> 9 rows selected (0.339 seconds)
> {code}
> IS DISTINCT FROM predicate returns correct result
> {code}
> select
> count(*)
> from
> j1 INNER JOIN j2 ON
> ( j1.c_double = j2.c_double)
> where
> j1.c_bigint IS DISTINCT FROM j2.c_bigint
> {code}
> Explain plan for query that returns incorrect result:
> {code}
> 00-01 StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-02Project($f0=[0])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[CAST(CASE(IS NULL($1), IS NULL($3), IS
> NULL($3), IS NULL($1), =($1, $3))):BOOLEAN NOT NULL])
> 00-05 HashJoin(condition=[=($0, $2)], joinType=[inner])
> 00-07Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j1]],
> selectionRoot=/joins/j1, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> 00-06Project(c_double0=[$0], c_bigint0=[$1])
> 00-08 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j2]],
> selectionRoot=/joins/j2, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2398) IS NOT DISTINCT FROM predicate returns incorrect result when used as a join filter
[
https://issues.apache.org/jira/browse/DRILL-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876027#comment-14876027
]
Victoria Markman commented on DRILL-2398:
-
Closing it, we have tests that verify this issue.
> IS NOT DISTINCT FROM predicate returns incorrect result when used as a join
> filter
> --
>
> Key: DRILL-2398
> URL: https://issues.apache.org/jira/browse/DRILL-2398
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: j1.parquet, j2.parquet
>
>
> count(*) should return 0 and not NULL
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > j1 INNER JOIN j2 ON
> . . . . . . . . . . . . > ( j1.c_double = j2.c_double)
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > j1.c_bigint IS NOT DISTINCT FROM j2.c_bigint
> . . . . . . . . . . . . > ;
> ++
> | EXPR$0 |
> ++
> ++
> {code}
> These are the values in the table
> {code}
> 0: jdbc:drill:schema=dfs> select j1.c_bigint, j2.c_bigint, count(*) from j1
> INNER JOIN j2 ON (j1.c_double = j2.c_double) group by j1.c_bigint,
> j2.c_bigint;
> ++++
> | c_bigint | c_bigint1 | EXPR$1 |
> ++++
> | 460194667 | -498749284 | 1 |
> | 464547172 | -498828740 | 1 |
> | 467451850 | -498966611 | 2 |
> | 471050029 | -499154096 | 3 |
> | 472873799 | -499233550 | 3 |
> | 475698977 | -499395929 | 2 |
> | 478986584 | -499564607 | 1 |
> | 488139464 | -499763274 | 3 |
> | 498214699 | -499871720 | 2 |
> ++++
> 9 rows selected (0.339 seconds)
> {code}
> IS DISTINCT FROM predicate returns correct result
> {code}
> select
> count(*)
> from
> j1 INNER JOIN j2 ON
> ( j1.c_double = j2.c_double)
> where
> j1.c_bigint IS DISTINCT FROM j2.c_bigint
> {code}
> Explain plan for query that returns incorrect result:
> {code}
> 00-01 StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-02Project($f0=[0])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[CAST(CASE(IS NULL($1), IS NULL($3), IS
> NULL($3), IS NULL($1), =($1, $3))):BOOLEAN NOT NULL])
> 00-05 HashJoin(condition=[=($0, $2)], joinType=[inner])
> 00-07Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j1]],
> selectionRoot=/joins/j1, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> 00-06Project(c_double0=[$0], c_bigint0=[$1])
> 00-08 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/joins/j2]],
> selectionRoot=/joins/j2, numFiles=1, columns=[`c_double`, `c_bigint`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3804) Some UNSUPPORTED messages are confusing to users
Victoria Markman created DRILL-3804:
---
Summary: Some UNSUPPORTED messages are confusing to users
Key: DRILL-3804
URL: https://issues.apache.org/jira/browse/DRILL-3804
Project: Apache Drill
Issue Type: Improvement
Components: Query Planning & Optimization
Reporter: Victoria Markman
Assignee: Jinfeng Ni
This error message includes JIRA that has been already fixed.
DRILL-1937 - is a request to print nice error message. Instead, it should be
pointing to the open JIRA with the feature that is not yet implemented.
{code}
0: jdbc:drill:schema=dfs> select * from t1 where a1 <> (select a2 from t2);
Error: UNSUPPORTED_OPERATION ERROR: Non-scalar sub-query used in an expression
See Apache Drill JIRA: DRILL-1937
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (DRILL-3258) Add support for non-scalar subqueries in expressions within HAVING clause
[
https://issues.apache.org/jira/browse/DRILL-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-3258:
-
> Add support for non-scalar subqueries in expressions within HAVING clause
> -
>
> Key: DRILL-3258
> URL: https://issues.apache.org/jira/browse/DRILL-3258
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Drill currently does not support queries with non-scalar sub-queries (which
> return more than one row) used within expressions.
> {code:sql}
> SELECT ss_item_sk
> FROM store_sales
> GROUP BY ss_item_sk
> HAVING Avg(ss_net_profit) > 2 *
> (SELECT avg(ss_net_profit)
> FROM store_sales
> GROUP BY ss_store_sk);
> {code}
> Such queries should be supported by Drill.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3258) Add support for non-scalar subqueries in expressions within HAVING clause
[
https://issues.apache.org/jira/browse/DRILL-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876094#comment-14876094
]
Victoria Markman commented on DRILL-3258:
-
I think this is a request to support non scalar subqueries and it should remain
open.
> Add support for non-scalar subqueries in expressions within HAVING clause
> -
>
> Key: DRILL-3258
> URL: https://issues.apache.org/jira/browse/DRILL-3258
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
>
> Drill currently does not support queries with non-scalar sub-queries (which
> return more than one row) used within expressions.
> {code:sql}
> SELECT ss_item_sk
> FROM store_sales
> GROUP BY ss_item_sk
> HAVING Avg(ss_net_profit) > 2 *
> (SELECT avg(ss_net_profit)
> FROM store_sales
> GROUP BY ss_store_sk);
> {code}
> Such queries should be supported by Drill.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3258) Add support for non-scalar subqueries in expressions within HAVING clause
[
https://issues.apache.org/jira/browse/DRILL-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3258:
Fix Version/s: (was: 1.2.0)
> Add support for non-scalar subqueries in expressions within HAVING clause
> -
>
> Key: DRILL-3258
> URL: https://issues.apache.org/jira/browse/DRILL-3258
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
>
> Drill currently does not support queries with non-scalar sub-queries (which
> return more than one row) used within expressions.
> {code:sql}
> SELECT ss_item_sk
> FROM store_sales
> GROUP BY ss_item_sk
> HAVING Avg(ss_net_profit) > 2 *
> (SELECT avg(ss_net_profit)
> FROM store_sales
> GROUP BY ss_store_sk);
> {code}
> Such queries should be supported by Drill.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3258) Add support for non-scalar subqueries in expressions within HAVING clause
[
https://issues.apache.org/jira/browse/DRILL-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876095#comment-14876095
]
Victoria Markman commented on DRILL-3258:
-
I'm removing target milestone: this can't be done by 1.2.0
> Add support for non-scalar subqueries in expressions within HAVING clause
> -
>
> Key: DRILL-3258
> URL: https://issues.apache.org/jira/browse/DRILL-3258
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Reporter: Abhishek Girish
>Assignee: Sean Hsuan-Yi Chu
>
> Drill currently does not support queries with non-scalar sub-queries (which
> return more than one row) used within expressions.
> {code:sql}
> SELECT ss_item_sk
> FROM store_sales
> GROUP BY ss_item_sk
> HAVING Avg(ss_net_profit) > 2 *
> (SELECT avg(ss_net_profit)
> FROM store_sales
> GROUP BY ss_store_sk);
> {code}
> Such queries should be supported by Drill.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3045:
Labels: document (was: )
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.j
[jira] [Commented] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876256#comment-14876256
]
Victoria Markman commented on DRILL-3045:
-
This bug introduces new configuration parameter: planner.memory_limit which
defines how much memory will be used by a query for planning purposes.
It is allocated from direct memory and if multiple queries are running
concurrently, each of them will allocate amount defined by this parameter.
Default value is 256M (268435456 bytes), range is between 1048576 and
17179869184 (1M - 16GB)
Notes about usage: if you notice that partition pruning is not taking place
(warning in drillbit.log or in the plan scan operator ALL the files appear
instead of one where filter is applied to) : increase the allocation and rerun
query again.
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
>
[jira] [Commented] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14876316#comment-14876316
]
Victoria Markman commented on DRILL-3045:
-
This fix was verified with 1.2.0
#Wed Sep 16 00:30:47 UTC 2015
git.commit.id.abbrev=9afcf61
Verification looked like this:
1. run query that queries file system with lots of files, and keep reducing
the size of `planner.memory_limit.` Queru that failed to prune for me used 16M
of planner memory
2. when query throws a warning and fails to prune, increased
planner.memory_limit to 32M, 64M, 128M, 256M
Need to find a way to automate this case in our framework. I consider it fixed.
Docs have been notified.
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCh
[jira] [Closed] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3045.
---
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-dril
[jira] [Resolved] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman resolved DRILL-3045.
-
Resolution: Fixed
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document, no_verified_test
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(Volca
[jira] [Closed] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3045.
---
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document, no_verified_test
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [o
[jira] [Reopened] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-3045:
-
Reopened for adding label
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document, no_verified_test
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(
[jira] [Updated] (DRILL-3045) Drill is not partition pruning due to internal off-heap memory limit for planning phase
[
https://issues.apache.org/jira/browse/DRILL-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3045:
Labels: document no_verified_test (was: document)
> Drill is not partition pruning due to internal off-heap memory limit for
> planning phase
> ---
>
> Key: DRILL-3045
> URL: https://issues.apache.org/jira/browse/DRILL-3045
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Mehant Baid
> Labels: document, no_verified_test
> Fix For: 1.2.0
>
> Attachments: DRILL-3045.patch
>
>
> The symptom is: we are running simple query of the form "select x from t
> where dir0='xyz and dir1='2015-01-01';" partition pruning works for a while
> and then it stops working.
> Query does run (since we don't fail the query in the case when we failed to
> prune) and return correct results.
> drillbit.log
> {code}
> 015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.exec.memory.BufferAllocator - Unable to allocate buffer of size 5000
> due to memory limit. Current allocation: 16776840
> java.lang.Exception: null
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:220)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:231)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:333)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:223)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:275)
> [optiq-core-0.9-drill-r20.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:206)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:145)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> 2015-04-19 15:54:22,027 [2acc305b-8f77-09af-1376-f6475c6a23c3:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.NullPointerException: null
> at
> org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:334)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:185)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:187)
> ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule$2.onMatch(PruneScanRule.java:110)
> [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at
> org.eigenbase.relopt.volcan
[jira] [Updated] (DRILL-3621) Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
[
https://issues.apache.org/jira/browse/DRILL-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3621:
Assignee: Khurram Faraaz (was: Jinfeng Ni)
> Wrong results when Drill on Hbase query contains rowkey "or" or "IN"
>
>
> Key: DRILL-3621
> URL: https://issues.apache.org/jira/browse/DRILL-3621
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Hao Zhu
>Assignee: Khurram Faraaz
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3621-Fix-incorrect-result-if-HBase-filter-cont.patch
>
>
> If Drill on Hbase query contains row_key "in" or "or", it produces wrong
> results.
> For example:
> 1. Create a hbase table
> {code}
> create 'testrowkey','cf'
> put 'testrowkey','DUMMY1','cf:c','value1'
> put 'testrowkey','DUMMY2','cf:c','value2'
> put 'testrowkey','DUMMY3','cf:c','value3'
> put 'testrowkey','DUMMY4','cf:c','value4'
> put 'testrowkey','DUMMY5','cf:c','value5'
> put 'testrowkey','DUMMY6','cf:c','value6'
> put 'testrowkey','DUMMY7','cf:c','value7'
> put 'testrowkey','DUMMY8','cf:c','value8'
> put 'testrowkey','DUMMY9','cf:c','value9'
> put 'testrowkey','DUMMY10','cf:c','value10'
> {code}
> 2. Drill queries:
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY10';
> +--+
> |RK|
> +--+
> | DUMMY10 |
> +--+
> 1 row selected (1.186 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY =
> 'DUMMY1';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.691 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.71 seconds)
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY
> ='DUMMY1' OR ROW_KEY = 'DUMMY10';
> +-+
> | RK|
> +-+
> | DUMMY1 |
> +-+
> 1 row selected (0.693 seconds)
> {code}
> From explain plan, filter is pushed down to hbase scan layer.
> {code}
> 0: jdbc:drill:zk=h2.poc.com:5181,h3.poc.com:5> explain plan for SELECT
> CONVERT_FROM(ROW_KEY,'UTF8') RK FROM hbase.testrowkey T WHERE ROW_KEY IN
> ('DUMMY1' , 'DUMMY10');
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(RK=[CONVERT_FROMUTF8($0)])
> 00-02Scan(groupscan=[HBaseGroupScan [HBaseScanSpec=HBaseScanSpec
> [tableName=testrowkey, startRow=DUMMY1, stopRow=DUMMY10, filter=null],
> columns=[`row_key`]]])
> |
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3463) Unit test of project pushdown in TestUnionAll should put more precisely plan attribute in plan verification.
[
https://issues.apache.org/jira/browse/DRILL-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901045#comment-14901045
]
Victoria Markman commented on DRILL-3463:
-
Verified changes in the expected result of the unit test are correct.
> Unit test of project pushdown in TestUnionAll should put more precisely plan
> attribute in plan verification.
> --
>
> Key: DRILL-3463
> URL: https://issues.apache.org/jira/browse/DRILL-3463
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3463-Unit-test-of-project-pushdown-in-TestUnio.patch
>
>
> As part of fix for DRILL-2802, it was discovered that several unit test cases
> for project pushdown in TestUnionAll did not put the desired plan attributes
> in to the expected plan result.
> To verify project pushdown is working properly, one simple way is to verify
> that the the column list in the Scan operator contains the desired columns.
> This should be the part of plan verification. However, the unit test cases in
> TestUnionAll did not do that. In stead, it tries to match a pattern of
> "Project -- Scan", which seems not serving the purpose it desired.
> For instance,
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*\n" +
> {code}
> should be replaced by
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*columns=\\[`n_comment`, `n_nationkey`, `n_name`\\].*\n"
> {code}
> if we want to verify the column 'n_comment', 'n_nationkey', 'n_name' are
> pushed into Scan operator.
> To fix this, modify the expected plan result, such that it contains the plan
> attributes that should be able to verify whether the project pushdown is
> working or not.
> This will help catch project pushdown failure, and avoid causing more false
> alarm in plan verification.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3463) Unit test of project pushdown in TestUnionAll should put more precisely plan attribute in plan verification.
[
https://issues.apache.org/jira/browse/DRILL-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3463.
---
> Unit test of project pushdown in TestUnionAll should put more precisely plan
> attribute in plan verification.
> --
>
> Key: DRILL-3463
> URL: https://issues.apache.org/jira/browse/DRILL-3463
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Jinfeng Ni
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-3463-Unit-test-of-project-pushdown-in-TestUnio.patch
>
>
> As part of fix for DRILL-2802, it was discovered that several unit test cases
> for project pushdown in TestUnionAll did not put the desired plan attributes
> in to the expected plan result.
> To verify project pushdown is working properly, one simple way is to verify
> that the the column list in the Scan operator contains the desired columns.
> This should be the part of plan verification. However, the unit test cases in
> TestUnionAll did not do that. In stead, it tries to match a pattern of
> "Project -- Scan", which seems not serving the purpose it desired.
> For instance,
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*\n" +
> {code}
> should be replaced by
> {code}
> final String[] expectedPlan = {"UnionAll.*\n." +
> "*Project.*\n" +
> ".*Scan.*columns=\\[`n_comment`, `n_nationkey`, `n_name`\\].*\n"
> {code}
> if we want to verify the column 'n_comment', 'n_nationkey', 'n_name' are
> pushed into Scan operator.
> To fix this, modify the expected plan result, such that it contains the plan
> attributes that should be able to verify whether the project pushdown is
> working or not.
> This will help catch project pushdown failure, and avoid causing more false
> alarm in plan verification.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2808) Substr function throws weird message when wrong arguments are passed in
[
https://issues.apache.org/jira/browse/DRILL-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901507#comment-14901507
]
Victoria Markman commented on DRILL-2808:
-
Still in 1.2.0, tried with:
#Sun Sep 20 15:30:07 UTC 2015
git.commit.id.abbrev=092903d
> Substr function throws weird message when wrong arguments are passed in
> ---
>
> Key: DRILL-2808
> URL: https://issues.apache.org/jira/browse/DRILL-2808
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jason Altekruse
> Fix For: 1.3.0
>
>
> #Wed Apr 15 17:37:25 EDT 2015
> git.commit.id.abbrev=cb47df0
> {code}
> 0: jdbc:drill:schema=dfs> select substr(0, 1, 'abc') from sys.options limit 1;
> Query failed: SYSTEM ERROR: Unexpected exception during fragment
> initialization: Internal error: Error while applying rule
> ReduceExpressionsRule[Project], args
> [rel#53990:ProjectRel.NONE.ANY([]).[](child=rel#53989:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> [1f7e0c35-fc65-4a2f-b668-12e9baf5c9b8 on atsqa4-133.qa.lab:31010]
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> 2015-04-16 22:04:32,029 [2acfce0f-8af4-9786-4799-9180b9bac219:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING -->
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> ReduceExpressionsRule[Project], args
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:211)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ReduceExpressionsRule[Project], args
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> at org.eigenbase.util.Util.newInternal(Util.java:750)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:246)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:278)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:196)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:137)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:155)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:770)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:202)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> ... 3 common frames omitted
> Caused by: java.lang.RuntimeException: Error in evaluating function of
> castBIGINT
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:323)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:147)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:47)
> ~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.expr.fn.interpreter.InterpreterEvaluator$EvalVisitor.visitFunctionHolderExpression(InterpreterEvaluator.java:251)
> ~[drill-java-ex
[jira] [Reopened] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-2967:
-
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: j2_j6_tables.tar, t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2 where b2 not in (select b1 from
> t1);
> ++++
> | a1 | b1 | c1 |
> ++++
> Query failed: SYSTEM ERROR: Join only supports implicit casts between 1.
> Numeric data
> 2. Varchar, Varbinary data Left type: INT, Right type: VARCHAR. Add explicit
> casts to avoid this error
> Fragment 1:0
> [1a83aa50-39aa-452c-91dd-970bf4a8f03d on atsqa4-133.qa.lab:31010]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing
> query.
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> at sqlline.SqlLine.print(SqlLine.java:1809)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
> at sqlline.SqlLine.dispatch(SqlLine.java:889)
> at sqlline.SqlLine.begin(SqlLine.java:763)
> at sqlline.SqlLine.start(SqlLine.java:498)
> at sqlline.SqlLine.main(SqlLine.java:460)
> {code}
> Explain plan for the query with an error:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select * from t2 where b2 not in
> (select b1 from t1);
> +++
> | text | json |
> +++
> | 00-00 Screen
> 00-01 Project(*=[$0])
> 00-02 UnionExchange
> 01-01 Project(T27¦¦*=[$0])
> 01-02 SelectionVectorRemover
> 01-03 Filter(condition=[NOT(CASE(=($2, 0), false, IS NOT NULL($6), true, IS
> NULL($4), null, <($3, $2), null, false))])
> 01-04 MergeJoin(condition=[=($4, $5)], joinType=[left])
> 01-06 SelectionVectorRemover
> 01-08 Sort(sort0=[$4], dir0=[ASC])
> 01-10 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4])
> 01-12 HashToRandomExchange(dist0=[[$4]])
> 02-01 UnorderedMuxExchange
> 04-01 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4],
> E_X_P_R_H_A_S_H_F_I_E_L_D=[castInt(hash64AsDouble($4))])
> 04-02 Project(T27¦¦*=[$0], b2=[$1], $f
[jira] [Commented] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902776#comment-14902776
]
Victoria Markman commented on DRILL-2967:
-
Reproducible with 1.2.0
#Sun Sep 20 15:30:07 UTC 2015
git.commit.id.abbrev=092903d
[~seanhychu] we've got to figure out the differences between your and my
environment.
{code}
0: jdbc:drill:schema=dfs> alter session set `planner.enable_hashjoin` = false;
+---+---+
| ok | summary |
+---+---+
| true | planner.enable_hashjoin updated. |
+---+---+
1 row selected (0.155 seconds)
0: jdbc:drill:schema=dfs> alter session set `planner.slice_target` = 1;
+---++
| ok |summary |
+---++
| true | planner.slice_target updated. |
+---++
1 row selected (0.154 seconds)
0: jdbc:drill:schema=dfs> select * from j2 where c_bigint not in ( select
cast(c_integer as bigint) from j6 where c_boolean is true ) and c_date not in (
select distinct c_date from j6);
java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
DrillRuntimeException: Join only supports implicit casts between 1. Numeric data
2. Varchar, Varbinary data 3. Date, Timestamp data Left type: DATE, Right
type: INT. Add explicit casts to avoid this error
Fragment 1:20
[Error Id: cac947b3-168f-4526-a7fa-c0e19311697d on atsqa4-133.qa.lab:31010]
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1583)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:738)
at sqlline.SqlLine.begin(SqlLine.java:612)
at sqlline.SqlLine.start(SqlLine.java:366)
at sqlline.SqlLine.main(SqlLine.java:259)
{code}
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: j2_j6_tables.tar, t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:dril
[jira] [Created] (DRILL-3823) Exception when grouping by case expression and columns array
Victoria Markman created DRILL-3823:
---
Summary: Exception when grouping by case expression and columns
array
Key: DRILL-3823
URL: https://issues.apache.org/jira/browse/DRILL-3823
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.1.0, 1.2.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni
In drill version 1.2.0:
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > count(*),
. . . . . . . . . . . . > case when columns[0] = '' then cast(null as
integer) else cast(columns[0] as integer) end
. . . . . . . . . . . . > from
. . . . . . . . . . . . > `t1.csv`
. . . . . . . . . . . . > group by
. . . . . . . . . . . . > case when columns[0] = '' then cast(null as
integer) else cast(columns[0] as integer) end;
Error: SYSTEM ERROR: UnsupportedOperationException: class
org.apache.calcite.sql.SqlLiteral: NULL
[Error Id: e4b37748-e81a-41c9-bfbc-e8f60cb43977 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
drillbit.log
{code}
2015-09-23 00:00:13,947 [29fe1472-1af8-ba17-4184-3bc38da1d372:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR:
UnsupportedOperationException: class org.apache.calcite.sql.SqlLiteral: NULL
[Error Id: 46bce8a5-048a-4caa-bcfc-9b2326cfd831 on atsqa4-133.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
UnsupportedOperationException: class org.apache.calcite.sql.SqlLiteral: NULL
[Error Id: 46bce8a5-048a-4caa-bcfc-9b2326cfd831 on atsqa4-133.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:524)
~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
exception during fragment initialization: Internal error: while converting CASE
WHEN `t1.csv`.`columns`[0] = '' THEN CAST(NULL AS INTEGER) ELSE
CAST(`t1.csv`.`columns`[0] AS INTEGER)
END
... 4 common frames omitted
Caused by: java.lang.AssertionError: Internal error: while converting CASE WHEN
`t1.csv`.`columns`[0] = '' THEN CAST(NULL AS INTEGER) ELSE
CAST(`t1.csv`.`columns`[0] AS INTEGER) END
at org.apache.calcite.util.Util.newInternal(Util.java:792)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.ReflectiveConvertletTable$1.convertCall(ReflectiveConvertletTable.java:96)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4162)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3595)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4054)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlToRelConverter$AggConverter.addGroupExpr(SqlToRelConverter.java:4346)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2432)
~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
at
org.apache.calcite.sql2rel.SqlToR
[jira] [Updated] (DRILL-3823) Exception when grouping by case expression and columns array
[
https://issues.apache.org/jira/browse/DRILL-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3823:
Attachment: t1.csv
> Exception when grouping by case expression and columns array
>
>
> Key: DRILL-3823
> URL: https://issues.apache.org/jira/browse/DRILL-3823
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Attachments: t1.csv
>
>
> In drill version 1.2.0:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*),
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as
> integer) else cast(columns[0] as integer) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > `t1.csv`
> . . . . . . . . . . . . > group by
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as
> integer) else cast(columns[0] as integer) end;
> Error: SYSTEM ERROR: UnsupportedOperationException: class
> org.apache.calcite.sql.SqlLiteral: NULL
> [Error Id: e4b37748-e81a-41c9-bfbc-e8f60cb43977 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> 2015-09-23 00:00:13,947 [29fe1472-1af8-ba17-4184-3bc38da1d372:foreman] ERROR
> o.a.drill.exec.work.foreman.Foreman - SYSTEM ERROR:
> UnsupportedOperationException: class org.apache.calcite.sql.SqlLiteral: NULL
> [Error Id: 46bce8a5-048a-4caa-bcfc-9b2326cfd831 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> UnsupportedOperationException: class org.apache.calcite.sql.SqlLiteral: NULL
> [Error Id: 46bce8a5-048a-4caa-bcfc-9b2326cfd831 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:524)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:742)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:841)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:786)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:788)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:894)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:255)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: Internal error: while converting
> CASE WHEN `t1.csv`.`columns`[0] = '' THEN CAST(NULL AS INTEGER) ELSE
> CAST(`t1.csv`.`columns`[0] AS INTEGER)
> END
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: while converting CASE
> WHEN `t1.csv`.`columns`[0] = '' THEN CAST(NULL AS INTEGER) ELSE
> CAST(`t1.csv`.`columns`[0] AS INTEGER) END
> at org.apache.calcite.util.Util.newInternal(Util.java:792)
> ~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$1.convertCall(ReflectiveConvertletTable.java:96)
> ~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4162)
> ~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3595)
> ~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
> ~[calcite-core-1.4.0-drill-r2.jar:1.4.0-drill-r2]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4054)
> ~[calcite-core-1.4
[jira] [Created] (DRILL-3827) Empty metadata file causes queries on the table to fail
Victoria Markman created DRILL-3827:
---
Summary: Empty metadata file causes queries on the table to fail
Key: DRILL-3827
URL: https://issues.apache.org/jira/browse/DRILL-3827
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.2.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni
Priority: Critical
I ran into a situation where drill created an empty metadata file (which is a
separate issue and I will try to narrow it down. Suspicion is that this happens
when "refresh table metada x" fails with "permission denied" error).
However, we need to guard against situation where metadata file is empty or
corrupted. We probably should skip reading it if we encounter unexpected result
and continue with query planning without that information. In the same fashion
as partition pruning failure. It's also important to log this information
somewhere, drillbit.log as a start. It would be really nice to have a flag in
the query profile that tells a user if we used metadata file for planning or
not. Will help in debugging performance issues.
Very confusing exception is thrown if you have zero length meta data file in
the directory:
{code}
[Wed Sep 23 07:45:28] # ls -la
total 2
drwxr-xr-x 2 root root 2 Sep 10 14:55 .
drwxr-xr-x 16 root root 35 Sep 15 12:54 ..
-rwxr-xr-x 1 root root 483 Jul 1 11:29 0_0_0.parquet
-rwxr-xr-x 1 root root 0 Sep 10 14:55 .drill.parquet_metadata
0: jdbc:drill:schema=dfs> select * from t1;
Error: SYSTEM ERROR: JsonMappingException: No content to map due to end-of-input
at [Source: com.mapr.fs.MapRFsDataInputStream@342bd88d; line: 1, column: 1]
[Error Id: c97574f6-b3e8-4183-8557-c30df6ca675f on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
Workaround is trivial, remove the file. Marking it as critical, since we don't
have any concurrency control in place and this file can get corrupted as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2748.
---
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2748) Filter is not pushed down into subquery with the group by
[
https://issues.apache.org/jira/browse/DRILL-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904614#comment-14904614
]
Victoria Markman commented on DRILL-2748:
-
Verified fixed in 1.2.0
#Tue Sep 22 19:46:29 UTC 2015
git.commit.id.abbrev=942d352
Added a test suite to test filter pushdown in isolation:
Functional/Passing/filter/pushdown
> Filter is not pushed down into subquery with the group by
> -
>
> Key: DRILL-2748
> URL: https://issues.apache.org/jira/browse/DRILL-2748
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0, 1.0.0, 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments:
> 0001-DRILL-2748-Improve-cost-estimation-for-Drill-logical.patch
>
>
> I'm not sure about this one, theoretically filter could have been pushed into
> the subquery.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from (select a1,
> b1, avg(a1) from t1 group by a1, b1) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[CAST(/(CastHigh(CASE(=($3, 0), null,
> $2)), $3)):ANY NOT NULL])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1}], agg#0=[$SUM0($0)],
> agg#1=[COUNT($0)])
> 00-06Project(a1=[$1], b1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`,
> `b1`]]])
> {code}
> Same with distinct in subquery:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select x, y, z from ( select
> distinct a1, b1, c1 from t1 ) as sq(x, y, z) where x = 10;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 Project(x=[$0], y=[$1], z=[$2])
> 00-02Project(x=[$0], y=[$1], z=[$2])
> 00-03 SelectionVectorRemover
> 00-04Filter(condition=[=($0, 10)])
> 00-05 HashAgg(group=[{0, 1, 2}])
> 00-06Project(a1=[$2], b1=[$1], c1=[$0])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/predicates/t1]],
> selectionRoot=/drill/testdata/predicates/t1, numFiles=1, columns=[`a1`, `b1`,
> `c1`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3470) Add support for multiple partition by clauses for window functions in the same query
[ https://issues.apache.org/jira/browse/DRILL-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman closed DRILL-3470. --- > Add support for multiple partition by clauses for window functions in the > same query > > > Key: DRILL-3470 > URL: https://issues.apache.org/jira/browse/DRILL-3470 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Jinfeng Ni > Labels: window_function > Fix For: 1.2.0 > > > We disabled support for multiple partitions in 1.1. (DRILL-3196) > This is a request to enable and productize multiple partitions for window > functions in the same sql statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3470) Add support for multiple partition by clauses for window functions in the same query
[ https://issues.apache.org/jira/browse/DRILL-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906516#comment-14906516 ] Victoria Markman commented on DRILL-3470: - Support added in 1.2.0 > Add support for multiple partition by clauses for window functions in the > same query > > > Key: DRILL-3470 > URL: https://issues.apache.org/jira/browse/DRILL-3470 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Jinfeng Ni > Labels: window_function > Fix For: 1.2.0 > > > We disabled support for multiple partitions in 1.1. (DRILL-3196) > This is a request to enable and productize multiple partitions for window > functions in the same sql statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
