[jira] [Updated] (DRILL-3359) Drill should throw and error when window function defined using WINDOW AS uses ROWS UNBOUNDED PRECEDING
[
https://issues.apache.org/jira/browse/DRILL-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3359:
Labels: doc-impacting no_verified_test window_function (was:
window_function)
> Drill should throw and error when window function defined using WINDOW AS
> uses ROWS UNBOUNDED PRECEDING
> ---
>
> Key: DRILL-3359
> URL: https://issues.apache.org/jira/browse/DRILL-3359
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Deneche A. Hakim
>Assignee: Sean Hsuan-Yi Chu
> Labels: doc-impacting, no_verified_test, window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3359.1.patch
>
>
> as part of DRILL-3188, the following query is not supported and Drill
> displays the proper error message:
> {noformat}
> 0: jdbc:drill:zk=local> select sum(salary) over(partition by position_id
> order by salary rows unbounded preceding) from cp.`employee.json` limit 20;
> Error: UNSUPPORTED_OPERATION ERROR: This type of window frame is currently
> not supported
> See Apache Drill JIRA: DRILL-3188
> {noformat}
> But when defining the same window using a WINDOW AS, Drill doesn't throw any
> error:
> {noformat}
> 0: jdbc:drill:zk=local> select sum(salary) over w from cp.`employee.json`
> window w as (partition by position_id order by salary rows unbounded
> preceding) limit 20;
> +---+
> | EXPR$0 |
> +---+
> | 8.0 |
> | 3.0 |
> | 135000.0 |
> | 135000.0 |
> | 135000.0 |
> | 215000.0 |
> | 215000.0 |
> | 25000.0 |
> | 15000.0 |
> | 5.0 |
> | 6700.0|
> | 14700.0 |
> | 34700.0 |
> | 34700.0 |
> | 5000.0|
> | 13500.0 |
> | 58500.0 |
> | 5000.0|
> | 11700.0 |
> | 2.0 |
> +---+
> 20 rows selected (0.348 seconds)
> {noformat}
> The results are, of course, incorrect
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3359) Drill should throw and error when window function defined using WINDOW AS uses ROWS UNBOUNDED PRECEDING
[
https://issues.apache.org/jira/browse/DRILL-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3359.
---
> Drill should throw and error when window function defined using WINDOW AS
> uses ROWS UNBOUNDED PRECEDING
> ---
>
> Key: DRILL-3359
> URL: https://issues.apache.org/jira/browse/DRILL-3359
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Deneche A. Hakim
>Assignee: Sean Hsuan-Yi Chu
> Labels: doc-impacting, no_verified_test, window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3359.1.patch
>
>
> as part of DRILL-3188, the following query is not supported and Drill
> displays the proper error message:
> {noformat}
> 0: jdbc:drill:zk=local> select sum(salary) over(partition by position_id
> order by salary rows unbounded preceding) from cp.`employee.json` limit 20;
> Error: UNSUPPORTED_OPERATION ERROR: This type of window frame is currently
> not supported
> See Apache Drill JIRA: DRILL-3188
> {noformat}
> But when defining the same window using a WINDOW AS, Drill doesn't throw any
> error:
> {noformat}
> 0: jdbc:drill:zk=local> select sum(salary) over w from cp.`employee.json`
> window w as (partition by position_id order by salary rows unbounded
> preceding) limit 20;
> +---+
> | EXPR$0 |
> +---+
> | 8.0 |
> | 3.0 |
> | 135000.0 |
> | 135000.0 |
> | 135000.0 |
> | 215000.0 |
> | 215000.0 |
> | 25000.0 |
> | 15000.0 |
> | 5.0 |
> | 6700.0|
> | 14700.0 |
> | 34700.0 |
> | 34700.0 |
> | 5000.0|
> | 13500.0 |
> | 58500.0 |
> | 5000.0|
> | 11700.0 |
> | 2.0 |
> +---+
> 20 rows selected (0.348 seconds)
> {noformat}
> The results are, of course, incorrect
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3359) Drill should throw and error when window function defined using WINDOW AS uses ROWS UNBOUNDED PRECEDING
[
https://issues.apache.org/jira/browse/DRILL-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14603281#comment-14603281
]
Victoria Markman commented on DRILL-3359:
-
My comment "ROWS" farm was disabled completely, should read: "These are the
cases where ROWS was disabled".
One case will be supported: when there is NO "ORDER BY" clause you can specify:
"ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING"
e.g. {code} select sum(salary) over(partition by position_id rows between
unbounded preceding and unbounded following) from cp.`employee.json`; {code}
> Drill should throw and error when window function defined using WINDOW AS
> uses ROWS UNBOUNDED PRECEDING
> ---
>
> Key: DRILL-3359
> URL: https://issues.apache.org/jira/browse/DRILL-3359
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Deneche A. Hakim
>Assignee: Sean Hsuan-Yi Chu
> Labels: doc-impacting, no_verified_test, window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3359.1.patch
>
>
> as part of DRILL-3188, the following query is not supported and Drill
> displays the proper error message:
> {noformat}
> 0: jdbc:drill:zk=local> select sum(salary) over(partition by position_id
> order by salary rows unbounded preceding) from cp.`employee.json` limit 20;
> Error: UNSUPPORTED_OPERATION ERROR: This type of window frame is currently
> not supported
> See Apache Drill JIRA: DRILL-3188
> {noformat}
> But when defining the same window using a WINDOW AS, Drill doesn't throw any
> error:
> {noformat}
> 0: jdbc:drill:zk=local> select sum(salary) over w from cp.`employee.json`
> window w as (partition by position_id order by salary rows unbounded
> preceding) limit 20;
> +---+
> | EXPR$0 |
> +---+
> | 8.0 |
> | 3.0 |
> | 135000.0 |
> | 135000.0 |
> | 135000.0 |
> | 215000.0 |
> | 215000.0 |
> | 25000.0 |
> | 15000.0 |
> | 5.0 |
> | 6700.0|
> | 14700.0 |
> | 34700.0 |
> | 34700.0 |
> | 5000.0|
> | 13500.0 |
> | 58500.0 |
> | 5000.0|
> | 11700.0 |
> | 2.0 |
> +---+
> 20 rows selected (0.348 seconds)
> {noformat}
> The results are, of course, incorrect
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3274) remove option 'window.enable'
[ https://issues.apache.org/jira/browse/DRILL-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-3274: Labels: window_function (was: ) > remove option 'window.enable' > - > > Key: DRILL-3274 > URL: https://issues.apache.org/jira/browse/DRILL-3274 > Project: Apache Drill > Issue Type: Bug >Reporter: Deneche A. Hakim >Assignee: Deneche A. Hakim > Labels: window_function > Fix For: 1.2.0 > > > as part of DRILL-3200 window functions will be enabled by default. We > shouldn't need to disable window functions so it's safe to remove the > 'window.enable' option -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3409) Specifying default frame explicitly results in an error
Victoria Markman created DRILL-3409:
---
Summary: Specifying default frame explicitly results in an error
Key: DRILL-3409
URL: https://issues.apache.org/jira/browse/DRILL-3409
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.1.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni
If I spell out default frame, I get an error:
{code}
0: jdbc:drill:schema=dfs> select c_bigint, min(c_double) over(partition by
c_bigint order by c_date, c_time nulls first range between unbounded preceding
and current row) from j9;
Error: PARSE ERROR: From line 1, column 95 to line 1, column 99: RANGE clause
cannot be used with compound ORDER BY clause
[Error Id: fe955fc0-bc0f-4588-bdc2-24defdc9390c on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
If I don't specify explicitly "default" frame as in the example above: query
parses and returns the same result as Postgres:
{code}
0: jdbc:drill:schema=dfs> explain plan for select c_bigint, min(c_double)
over(partition by c_bigint order by c_date, c_time nulls first) from j9;
00-00Screen
00-01 ProjectAllowDup(c_bigint=[$0], EXPR$1=[$1])
00-02Project(c_bigint=[$1], w0$o0=[$5])
00-03 Window(window#0=[window(partition {1} order by [3, 4
ASC-nulls-first] range between UNBOUNDED PRECEDING and CURRENT ROW aggs
[MIN($2)])])
00-04SelectionVectorRemover
00-05 Sort(sort0=[$1], sort1=[$3], sort2=[$4], dir0=[ASC],
dir1=[ASC], dir2=[ASC-nulls-first])
00-06Project(T32¦¦*=[$0], c_bigint=[$1], c_double=[$2],
c_date=[$3], c_time=[$4])
00-07 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/j9]],
selectionRoot=/drill/testdata/subqueries/j9, numFiles=1, columns=[`*`]]])
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3409) Specifying default frame explicitly results in an error
[
https://issues.apache.org/jira/browse/DRILL-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3409:
Labels: window_function (was: )
> Specifying default frame explicitly results in an error
> ---
>
> Key: DRILL-3409
> URL: https://issues.apache.org/jira/browse/DRILL-3409
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Labels: window_function
>
> If I spell out default frame, I get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select c_bigint, min(c_double) over(partition by
> c_bigint order by c_date, c_time nulls first range between unbounded
> preceding and current row) from j9;
> Error: PARSE ERROR: From line 1, column 95 to line 1, column 99: RANGE clause
> cannot be used with compound ORDER BY clause
> [Error Id: fe955fc0-bc0f-4588-bdc2-24defdc9390c on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> If I don't specify explicitly "default" frame as in the example above: query
> parses and returns the same result as Postgres:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select c_bigint, min(c_double)
> over(partition by c_bigint order by c_date, c_time nulls first) from j9;
> 00-00Screen
> 00-01 ProjectAllowDup(c_bigint=[$0], EXPR$1=[$1])
> 00-02Project(c_bigint=[$1], w0$o0=[$5])
> 00-03 Window(window#0=[window(partition {1} order by [3, 4
> ASC-nulls-first] range between UNBOUNDED PRECEDING and CURRENT ROW aggs
> [MIN($2)])])
> 00-04SelectionVectorRemover
> 00-05 Sort(sort0=[$1], sort1=[$3], sort2=[$4], dir0=[ASC],
> dir1=[ASC], dir2=[ASC-nulls-first])
> 00-06Project(T32¦¦*=[$0], c_bigint=[$1], c_double=[$2],
> c_date=[$3], c_time=[$4])
> 00-07 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/j9]],
> selectionRoot=/drill/testdata/subqueries/j9, numFiles=1, columns=[`*`]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3238) Cannot Plan Exception is raised when the same window partition is defined in select & window clauses
[
https://issues.apache.org/jira/browse/DRILL-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568055#comment-14568055
]
Victoria Markman edited comment on DRILL-3238 at 6/27/15 4:15 PM:
--
Interestingly, this case "over W" works:
{code}
select sum(a2) over w, count(*) over(partition by a2 order by a2) from t2
window w as (partition by a2 order by a2);
{code}
I did not realize that over(W) is the supported grammar ...
was (Author: vicky):
Interestingly, this case "over W" works:
{code}
select sum(a2) over w, count(*) over(partition by a2 order by a2) from t2
window w as (partition by a2 order by a2);
{code}
I did not realized that over(W) is the supported grammar ...
> Cannot Plan Exception is raised when the same window partition is defined in
> select & window clauses
>
>
> Key: DRILL-3238
> URL: https://issues.apache.org/jira/browse/DRILL-3238
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Sean Hsuan-Yi Chu
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: Future
>
>
> While this works:
> {code}
> select sum(a2) over(partition by a2 order by a2), count(*) over(partition by
> a2 order by a2)
> from t
> {code}
> , this fails
> {code}
> select sum(a2) over(w), count(*) over(partition by a2 order by a2)
> from t
> window w as (partition by a2 order by a2)
> {code}
> Notice these two queries are logically the same thing if we plug-in the
> window definition back into the SELECT-CLAUSE in the 2nd query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3374) CTAS with PARTITION BY, partition column name from view can not be resolved
[
https://issues.apache.org/jira/browse/DRILL-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3374:
Labels: (was: window_function)
> CTAS with PARTITION BY, partition column name from view can not be resolved
> ---
>
> Key: DRILL-3374
> URL: https://issues.apache.org/jira/browse/DRILL-3374
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.1.0
>Reporter: Khurram Faraaz
>Assignee: Jinfeng Ni
> Fix For: 1.1.0
>
>
> CTAS with PARTITION BY clause fails to resolve column name when partitioning
> column is from a view.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> create table ctas_prtng_01 partition by
> (col_vchar_52) as select * from vwOnParq_wCst;
> Error: SYSTEM ERROR: IllegalArgumentException: partition col col_vchar_52
> could not be resolved in table's column lists!
> [Error Id: 7cb227c1-65c5-48cb-a00b-1a89a5309bc8 on centos-04.qa.lab:31010]
> (state=,code=0)
> {code}
> Table used in above CTAS does exist and the column used to partition by also
> exists.
> {code}
> 0: jdbc:drill:schema=dfs.tmp> describe vwOnParq_wCst;
> +---++--+
> | COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
> +---++--+
> | col_int | INTEGER| YES |
> | col_bigint| BIGINT | YES |
> | col_char_2| CHARACTER | YES |
> | col_vchar_52 | CHARACTER VARYING | YES |
> | col_tmstmp| TIMESTAMP | YES |
> | col_dt| DATE | YES |
> | col_booln | BOOLEAN| YES |
> | col_dbl | DOUBLE | YES |
> | col_tm| TIME | YES |
> +---++--+
> 9 rows selected (0.411 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3404) Filter on window function does not appear in query plan
[
https://issues.apache.org/jira/browse/DRILL-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3404:
Labels: window_function (was: )
> Filter on window function does not appear in query plan
> ---
>
> Key: DRILL-3404
> URL: https://issues.apache.org/jira/browse/DRILL-3404
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_function
> Attachments: 0_0_0.parquet
>
>
> Filter is missing in the query plan for the below query in Drill, and hence
> wrong results are returned.
> Results from Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select c1, c2, w_sum from ( select c1, c2, sum
> ( c1 ) over ( partition by c2 order by c1 asc nulls first ) w_sum from
> `tblWnulls` ) sub_query where w_sum is not null;
> +-+---+-+
> | c1 | c2 |w_sum|
> +-+---+-+
> | 0 | a | 0 |
> | 1 | a | 1 |
> | 5 | a | 6 |
> | 10 | a | 16 |
> | 11 | a | 27 |
> | 14 | a | 41 |
> | 1 | a | 11152 |
> | 2 | b | 2 |
> | 9 | b | 11 |
> | 13 | b | 24 |
> | 17 | b | 41 |
> | null| c | null|
> | 4 | c | 4 |
> | 6 | c | 10 |
> | 8 | c | 18 |
> | 12 | c | 30 |
> | 13 | c | 56 |
> | 13 | c | 56 |
> | null| d | null|
> | null| d | null|
> | 10 | d | 10 |
> | 11 | d | 21 |
> | 2147483647 | d | 4294967315 |
> | 2147483647 | d | 4294967315 |
> | -1 | e | -1 |
> | 15 | e | 14 |
> | null| null | null|
> | 19 | null | 19 |
> | 65536 | null | 6 |
> | 100 | null | 106 |
> +-+---+-+
> 30 rows selected (0.337 seconds)
> {code}
> Explain plan for the above query from Drill
> {code}
> 0: jdbc:drill:schema=dfs.tmp> explain plan for select c1, c2, w_sum from (
> select c1, c2, sum ( c1 ) over ( partition by c2 order by c1 asc nulls first
> ) w_sum from `tblWnulls` ) sub_query where w_sum is not null;
> +--+---+
> |
>
>
>
> text
>
>
>
>| json |
> +--+---+
> | 00-00Screen
> 00-01 Project(c1=[$0], c2=[$1], w_sum=[$2])
> 00-02Project(c1
[jira] [Commented] (DRILL-3298) Wrong result with SUM window function and order by without partition by in the OVER clause
[
https://issues.apache.org/jira/browse/DRILL-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605890#comment-14605890
]
Victoria Markman commented on DRILL-3298:
-
Verified fixed in: 1.1
Test: Passing/window_functions/bugs/drill-3298.sql
> Wrong result with SUM window function and order by without partition by in
> the OVER clause
> --
>
> Key: DRILL-3298
> URL: https://issues.apache.org/jira/browse/DRILL-3298
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3298.1.patch.txt, j1.tar, test.res
>
>
> This query returns incorrect result when planner.slice_target = 1
> {code}
> select
> j1.c_integer,
> sum(j1.c_integer) over w
> from j1
> window w as (order by c_integer desc)
> order by
> 1, 2;
> {code}
> Query plan with planner.slice_target = 1
> {noformat}
> 00-01 Project(c_integer=[$0], EXPR$1=[$1])
> 00-02SingleMergeExchange(sort0=[0 ASC], sort1=[1 ASC])
> 01-01 SelectionVectorRemover
> 01-02Sort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 01-03 Project(c_integer=[$0], EXPR$1=[$1])
> 01-04HashToRandomExchange(dist0=[[$0]], dist1=[[$1]])
> 02-01 UnorderedMuxExchange
> 03-01Project(c_integer=[$0], EXPR$1=[$1],
> E_X_P_R_H_A_S_H_F_I_E_L_D=[castInt(hash64AsDouble($1, hash64AsDouble($0)))])
> 03-02 Project(c_integer=[$0], EXPR$1=[CASE(>($1, 0),
> CAST($2):ANY, null)])
> 03-03Window(window#0=[window(partition {} order by [0
> DESC] range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT($0),
> $SUM0($0)])])
> 03-04 SelectionVectorRemover
> 03-05Sort(sort0=[$0], dir0=[DESC])
> 03-06 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/j1]],
> selectionRoot=/drill/testdata/subqueries/j1, numFiles=1,
> columns=[`c_integer`]]])
> {noformat}
> Query plan with planner.slice_target = 10;
> {noformat}
> 00-01 Project(c_integer=[$0], EXPR$1=[$1])
> 00-02SelectionVectorRemover
> 00-03 Sort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-04Project(c_integer=[$0], EXPR$1=[CASE(>($1, 0), CAST($2):ANY,
> null)])
> 00-05 Window(window#0=[window(partition {} order by [0 DESC]
> range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT($0),
> $SUM0($0)])])
> 00-06SelectionVectorRemover
> 00-07 Sort(sort0=[$0], dir0=[DESC])
> 00-08Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/j1]],
> selectionRoot=/drill/testdata/subqueries/j1, numFiles=1,
> columns=[`c_integer`]]])
> {noformat}
> Attached:
> * table j1
> * test.res - result generated with postgres
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3298) Wrong result with SUM window function and order by without partition by in the OVER clause
[
https://issues.apache.org/jira/browse/DRILL-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3298.
---
> Wrong result with SUM window function and order by without partition by in
> the OVER clause
> --
>
> Key: DRILL-3298
> URL: https://issues.apache.org/jira/browse/DRILL-3298
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Aman Sinha
>Priority: Critical
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3298.1.patch.txt, j1.tar, test.res
>
>
> This query returns incorrect result when planner.slice_target = 1
> {code}
> select
> j1.c_integer,
> sum(j1.c_integer) over w
> from j1
> window w as (order by c_integer desc)
> order by
> 1, 2;
> {code}
> Query plan with planner.slice_target = 1
> {noformat}
> 00-01 Project(c_integer=[$0], EXPR$1=[$1])
> 00-02SingleMergeExchange(sort0=[0 ASC], sort1=[1 ASC])
> 01-01 SelectionVectorRemover
> 01-02Sort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 01-03 Project(c_integer=[$0], EXPR$1=[$1])
> 01-04HashToRandomExchange(dist0=[[$0]], dist1=[[$1]])
> 02-01 UnorderedMuxExchange
> 03-01Project(c_integer=[$0], EXPR$1=[$1],
> E_X_P_R_H_A_S_H_F_I_E_L_D=[castInt(hash64AsDouble($1, hash64AsDouble($0)))])
> 03-02 Project(c_integer=[$0], EXPR$1=[CASE(>($1, 0),
> CAST($2):ANY, null)])
> 03-03Window(window#0=[window(partition {} order by [0
> DESC] range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT($0),
> $SUM0($0)])])
> 03-04 SelectionVectorRemover
> 03-05Sort(sort0=[$0], dir0=[DESC])
> 03-06 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/j1]],
> selectionRoot=/drill/testdata/subqueries/j1, numFiles=1,
> columns=[`c_integer`]]])
> {noformat}
> Query plan with planner.slice_target = 10;
> {noformat}
> 00-01 Project(c_integer=[$0], EXPR$1=[$1])
> 00-02SelectionVectorRemover
> 00-03 Sort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-04Project(c_integer=[$0], EXPR$1=[CASE(>($1, 0), CAST($2):ANY,
> null)])
> 00-05 Window(window#0=[window(partition {} order by [0 DESC]
> range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT($0),
> $SUM0($0)])])
> 00-06SelectionVectorRemover
> 00-07 Sort(sort0=[$0], dir0=[DESC])
> 00-08Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/j1]],
> selectionRoot=/drill/testdata/subqueries/j1, numFiles=1,
> columns=[`c_integer`]]])
> {noformat}
> Attached:
> * table j1
> * test.res - result generated with postgres
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
Victoria Markman created DRILL-3414:
---
Summary: Window function on a null producing column of an outer
join results in the wrong result
Key: DRILL-3414
URL: https://issues.apache.org/jira/browse/DRILL-3414
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.0.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni
Priority: Critical
{code:sql}
select
j4.c_boolean,
j4.c_date,
j4.c_integer,
sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
j4.c_integer)
fromj1
left outer join
j4 on j1.c_integer = j4.c_integer
order by 1,2,3;
{code}
If window function is on left side, query returns correct result.
This works:
{code:sql}
select
j1.c_boolean,
j1.c_date,
sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
from
j1
left outer join
j4 on j1.c_integer = j4.c_integer
order by
1, 2;
{code}
Attached:
1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
[
https://issues.apache.org/jira/browse/DRILL-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3414:
Labels: window_funcion (was: )
> Window function on a null producing column of an outer join results in the
> wrong result
> ---
>
> Key: DRILL-3414
> URL: https://issues.apache.org/jira/browse/DRILL-3414
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_funcion
> Attachments: j1.tar, j4.parquet, query.tar
>
>
> {code:sql}
> select
> j4.c_boolean,
> j4.c_date,
> j4.c_integer,
> sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
> j4.c_integer)
> fromj1
>left outer join
>j4 on j1.c_integer = j4.c_integer
> order by 1,2,3;
> {code}
> If window function is on left side, query returns correct result.
> This works:
> {code:sql}
> select
> j1.c_boolean,
> j1.c_date,
> sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
> from
> j1
> left outer join
> j4 on j1.c_integer = j4.c_integer
> order by
> 1, 2;
> {code}
> Attached:
> 1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
> 2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
[
https://issues.apache.org/jira/browse/DRILL-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3414:
Attachment: j4.parquet
j1.tar
query.tar
> Window function on a null producing column of an outer join results in the
> wrong result
> ---
>
> Key: DRILL-3414
> URL: https://issues.apache.org/jira/browse/DRILL-3414
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_funcion
> Attachments: j1.tar, j4.parquet, query.tar
>
>
> {code:sql}
> select
> j4.c_boolean,
> j4.c_date,
> j4.c_integer,
> sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
> j4.c_integer)
> fromj1
>left outer join
>j4 on j1.c_integer = j4.c_integer
> order by 1,2,3;
> {code}
> If window function is on left side, query returns correct result.
> This works:
> {code:sql}
> select
> j1.c_boolean,
> j1.c_date,
> sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
> from
> j1
> left outer join
> j4 on j1.c_integer = j4.c_integer
> order by
> 1, 2;
> {code}
> Attached:
> 1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
> 2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
[
https://issues.apache.org/jira/browse/DRILL-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605985#comment-14605985
]
Victoria Markman commented on DRILL-3414:
-
I'm working on a smaller reproduction.
> Window function on a null producing column of an outer join results in the
> wrong result
> ---
>
> Key: DRILL-3414
> URL: https://issues.apache.org/jira/browse/DRILL-3414
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_funcion
> Attachments: j1.tar, j4.parquet, query.tar
>
>
> {code:sql}
> select
> j4.c_boolean,
> j4.c_date,
> j4.c_integer,
> sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
> j4.c_integer)
> fromj1
>left outer join
>j4 on j1.c_integer = j4.c_integer
> order by 1,2,3;
> {code}
> If window function is on left side, query returns correct result.
> This works:
> {code:sql}
> select
> j1.c_boolean,
> j1.c_date,
> sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
> from
> j1
> left outer join
> j4 on j1.c_integer = j4.c_integer
> order by
> 1, 2;
> {code}
> Attached:
> 1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
> 2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
[
https://issues.apache.org/jira/browse/DRILL-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3414.
---
> Window function on a null producing column of an outer join results in the
> wrong result
> ---
>
> Key: DRILL-3414
> URL: https://issues.apache.org/jira/browse/DRILL-3414
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_funcion
> Attachments: j1.tar, j4.parquet, query.tar
>
>
> {code:sql}
> select
> j4.c_boolean,
> j4.c_date,
> j4.c_integer,
> sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
> j4.c_integer)
> fromj1
>left outer join
>j4 on j1.c_integer = j4.c_integer
> order by 1,2,3;
> {code}
> If window function is on left side, query returns correct result.
> This works:
> {code:sql}
> select
> j1.c_boolean,
> j1.c_date,
> sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
> from
> j1
> left outer join
> j4 on j1.c_integer = j4.c_integer
> order by
> 1, 2;
> {code}
> Attached:
> 1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
> 2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
[
https://issues.apache.org/jira/browse/DRILL-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606426#comment-14606426
]
Victoria Markman commented on DRILL-3414:
-
Jinfeng,
This is not a bug. It is my mistake. Postgres table somehow (still need to
figure out how this happened) had one row with NULL value !!!
Ship it :)
Closing as invalid. Yoo-hoo ...
Vicky.
> Window function on a null producing column of an outer join results in the
> wrong result
> ---
>
> Key: DRILL-3414
> URL: https://issues.apache.org/jira/browse/DRILL-3414
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_funcion
> Attachments: j1.tar, j4.parquet, query.tar
>
>
> {code:sql}
> select
> j4.c_boolean,
> j4.c_date,
> j4.c_integer,
> sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
> j4.c_integer)
> fromj1
>left outer join
>j4 on j1.c_integer = j4.c_integer
> order by 1,2,3;
> {code}
> If window function is on left side, query returns correct result.
> This works:
> {code:sql}
> select
> j1.c_boolean,
> j1.c_date,
> sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
> from
> j1
> left outer join
> j4 on j1.c_integer = j4.c_integer
> order by
> 1, 2;
> {code}
> Attached:
> 1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
> 2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-3414) Window function on a null producing column of an outer join results in the wrong result
[
https://issues.apache.org/jira/browse/DRILL-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman resolved DRILL-3414.
-
Resolution: Invalid
> Window function on a null producing column of an outer join results in the
> wrong result
> ---
>
> Key: DRILL-3414
> URL: https://issues.apache.org/jira/browse/DRILL-3414
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Labels: window_funcion
> Attachments: j1.tar, j4.parquet, query.tar
>
>
> {code:sql}
> select
> j4.c_boolean,
> j4.c_date,
> j4.c_integer,
> sum(j4.c_integer) over (partition by j4.c_boolean order by j4.c_date,
> j4.c_integer)
> fromj1
>left outer join
>j4 on j1.c_integer = j4.c_integer
> order by 1,2,3;
> {code}
> If window function is on left side, query returns correct result.
> This works:
> {code:sql}
> select
> j1.c_boolean,
> j1.c_date,
> sum(j1.c_integer) over (partition by j1.c_boolean order by j1.c_date)
> from
> j1
> left outer join
> j4 on j1.c_integer = j4.c_integer
> order by
> 1, 2;
> {code}
> Attached:
> 1. query.tar (q2.sql , q2.res (postgres output), q2.out (drill output) )
> 2. tables : j1.tar, j4.parquet
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3420) Final sort can be dropped in some cases when result of window operator is already sorted on the same columns
Victoria Markman created DRILL-3420:
---
Summary: Final sort can be dropped in some cases when result of
window operator is already sorted on the same columns
Key: DRILL-3420
URL: https://issues.apache.org/jira/browse/DRILL-3420
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.0.0
Reporter: Victoria Markman
Assignee: Jinfeng Ni
Fix For: 1.2.0
In the example query, output of the window operator is already sorted on the
same columns that are specified in "order by clause" Last sort is redundant.
{code}
0: jdbc:drill:schema=dfs> explain plan for select b1, c1, a1, sum(a1)
over(partition by b1 order by c1, a1) from t1 order by 1,2,3;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(b1=[$0], c1=[$1], a1=[$2], EXPR$3=[$3])
00-02SelectionVectorRemover
00-03 Sort(sort0=[$0], sort1=[$1], sort2=[$2], dir0=[ASC], dir1=[ASC],
dir2=[ASC])
00-04Project(b1=[$0], c1=[$1], a1=[$2], EXPR$3=[CASE(>($3, 0), $4,
null)])
00-05 Window(window#0=[window(partition {0} order by [1, 2] range
between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT($2), $SUM0($2)])])
00-06SelectionVectorRemover
00-07 Sort(sort0=[$0], sort1=[$1], sort2=[$2], dir0=[ASC],
dir1=[ASC], dir2=[ASC])
00-08Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/t1]],
selectionRoot=/drill/testdata/subqueries/t1, numFiles=1, columns=[`b1`, `c1`,
`a1`]]])
{code}
Note to QA: when this enhancement is implemented, we need to make sure that we
have cases where sort order is destroyed by subsequent operation on top of
window. In these cases "sort should still be planned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3420) Final sort can be dropped in some cases when result of window operator is already sorted on the same columns
[
https://issues.apache.org/jira/browse/DRILL-3420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3420:
Labels: window_function (was: )
> Final sort can be dropped in some cases when result of window operator is
> already sorted on the same columns
>
>
> Key: DRILL-3420
> URL: https://issues.apache.org/jira/browse/DRILL-3420
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Labels: window_function
> Fix For: 1.2.0
>
>
> In the example query, output of the window operator is already sorted on the
> same columns that are specified in "order by clause" Last sort is redundant.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select b1, c1, a1, sum(a1)
> over(partition by b1 order by c1, a1) from t1 order by 1,2,3;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(b1=[$0], c1=[$1], a1=[$2], EXPR$3=[$3])
> 00-02SelectionVectorRemover
> 00-03 Sort(sort0=[$0], sort1=[$1], sort2=[$2], dir0=[ASC],
> dir1=[ASC], dir2=[ASC])
> 00-04Project(b1=[$0], c1=[$1], a1=[$2], EXPR$3=[CASE(>($3, 0),
> $4, null)])
> 00-05 Window(window#0=[window(partition {0} order by [1, 2]
> range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT($2),
> $SUM0($2)])])
> 00-06SelectionVectorRemover
> 00-07 Sort(sort0=[$0], sort1=[$1], sort2=[$2], dir0=[ASC],
> dir1=[ASC], dir2=[ASC])
> 00-08Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/t1]],
> selectionRoot=/drill/testdata/subqueries/t1, numFiles=1, columns=[`b1`, `c1`,
> `a1`]]])
> {code}
> Note to QA: when this enhancement is implemented, we need to make sure that
> we have cases where sort order is destroyed by subsequent operation on top of
> window. In these cases "sort should still be planned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3211) Assert in a query with window function and group by clause
[
https://issues.apache.org/jira/browse/DRILL-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606578#comment-14606578
]
Victoria Markman commented on DRILL-3211:
-
Verified fixed in 1.1
#Mon Jun 29 17:55:24 EDT 2015
git.commit.id.abbrev=e923ac5
{code}
0: jdbc:drill:schema=dfs> select sum(a1) over (partition by b1) from t1 group
by b1;
Error: PARSE ERROR: From line 1, column 12 to line 1, column 13: Expression
'a1' is not being grouped
[Error Id: 284ab6f7-e7ee-4959-873d-8d18201d35de on atsqa4-133.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs> select sum(a2), rank() over(partition by b2 order by
a2) from t2 group by b2;
Error: PARSE ERROR: From line 1, column 54 to line 1, column 55: Expression
'a2' is not being grouped
[Error Id: f38dca9d-46c6-4f51-bcc6-250b01d83a22 on atsqa4-133.qa.lab:31010]
(state=,code=0)
0: jdbc:drill:schema=dfs> select sum(a2), row_number() over(partition by c2, b2
order by a2) from t2 group by c2,b2;
Error: PARSE ERROR: From line 1, column 64 to line 1, column 65: Expression
'a2' is not being grouped
[Error Id: ec1e065f-20ff-4bf4-bb2c-2af338db9194 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
> Assert in a query with window function and group by clause
> ---
>
> Key: DRILL-3211
> URL: https://issues.apache.org/jira/browse/DRILL-3211
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: window_function
> Fix For: 1.1.0
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select sum(a1) over (partition by b1) from t1
> group by b1;
> Error: SYSTEM ERROR: java.lang.AssertionError: Internal error: while
> converting SUM(`t1`.`a1`)
> [Error Id: 21872cfa-6f09-4e92-aee6-5dd8698cf9e7 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> Caused by: java.lang.AssertionError: Internal error: while converting
> SUM(`t1`.`a1`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:152)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertOver(SqlToRelConverter.java:1762)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1000(SqlToRelConverter.java:180)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3937)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2521)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2342)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:604)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:564)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2741)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:522)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.prepare.PlannerImpl.convert(PlannerImpl.java:198)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:246)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:182)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:177)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:902)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:240)
> [drill-java-exec
[jira] [Updated] (DRILL-3211) Assert in a query with window function and group by clause
[
https://issues.apache.org/jira/browse/DRILL-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3211:
Labels: no_verified_test window_function (was: window_function)
> Assert in a query with window function and group by clause
> ---
>
> Key: DRILL-3211
> URL: https://issues.apache.org/jira/browse/DRILL-3211
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: no_verified_test, window_function
> Fix For: 1.1.0
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select sum(a1) over (partition by b1) from t1
> group by b1;
> Error: SYSTEM ERROR: java.lang.AssertionError: Internal error: while
> converting SUM(`t1`.`a1`)
> [Error Id: 21872cfa-6f09-4e92-aee6-5dd8698cf9e7 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> Caused by: java.lang.AssertionError: Internal error: while converting
> SUM(`t1`.`a1`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:152)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertOver(SqlToRelConverter.java:1762)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1000(SqlToRelConverter.java:180)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3937)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2521)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2342)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:604)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:564)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2741)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:522)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.prepare.PlannerImpl.convert(PlannerImpl.java:198)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:246)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:182)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:177)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:902)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:240)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
> ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_71]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:142)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> ... 19 common frames omitted
> Caused by: java.lang.AssertionError: null
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.getRootField(SqlToRelConverter.java:3810)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.adjustInputRef(SqlToRelConverter.java:3139)
> ~[calcite-c
[jira] [Closed] (DRILL-3211) Assert in a query with window function and group by clause
[
https://issues.apache.org/jira/browse/DRILL-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3211.
---
> Assert in a query with window function and group by clause
> ---
>
> Key: DRILL-3211
> URL: https://issues.apache.org/jira/browse/DRILL-3211
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Labels: no_verified_test, window_function
> Fix For: 1.1.0
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select sum(a1) over (partition by b1) from t1
> group by b1;
> Error: SYSTEM ERROR: java.lang.AssertionError: Internal error: while
> converting SUM(`t1`.`a1`)
> [Error Id: 21872cfa-6f09-4e92-aee6-5dd8698cf9e7 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> Caused by: java.lang.AssertionError: Internal error: while converting
> SUM(`t1`.`a1`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:152)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertOver(SqlToRelConverter.java:1762)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1000(SqlToRelConverter.java:180)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3937)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2521)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2342)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:604)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:564)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:2741)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:522)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.prepare.PlannerImpl.convert(PlannerImpl.java:198)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRel(DefaultSqlHandler.java:246)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:182)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:177)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:902)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:240)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
> ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_71]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:142)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> ... 19 common frames omitted
> Caused by: java.lang.AssertionError: null
> at
> org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.getRootField(SqlToRelConverter.java:3810)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.adjustInputRef(SqlToRelConverter.java:3139)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.calci
[jira] [Updated] (DRILL-3344) When Group By clause is present, the argument in window function should not refer to any column outside Group By
[
https://issues.apache.org/jira/browse/DRILL-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3344:
Labels: no_verified_test window_function (was: window_function)
> When Group By clause is present, the argument in window function should not
> refer to any column outside Group By
>
>
> Key: DRILL-3344
> URL: https://issues.apache.org/jira/browse/DRILL-3344
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.1.0
> Environment: 6ebfbb9d0fc0b87b032f5e5d5cb0825f5464426e
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: no_verified_test, window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3344.1.patch, forPrqView.csv
>
>
> CTAS
> {code}
> 0: jdbc:drill:schema=dfs.tmp> create table tblForView(col_int, col_bigint,
> col_char_2, col_vchar_52, col_tmstmp, col_dt, col_booln, col_dbl, col_tm) as
> select cast(columns[0] as INT), cast(columns[1] as BIGINT),cast(columns[2] as
> CHAR(2)), cast(columns[3] as VARCHAR(52)), cast(columns[4] as TIMESTAMP),
> cast(columns[5] as DATE), cast(columns[6] as BOOLEAN),cast(columns[7] as
> DOUBLE),cast(columns[8] as TIME) from `forPrqView.csv`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 30 |
> +---++
> 1 row selected (0.586 seconds)
> {code}
> Failing query
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select max(col_tm) over(), col_char_2 from
> tblForView group by col_char_2;
> Error: SYSTEM ERROR: java.lang.AssertionError: Internal error: while
> converting MAX(`tblForView`.`col_tm`)
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace
> {code}
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> java.lang.AssertionError: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
> ~[drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:840)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:782)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:784)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:893)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:152)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertOver(SqlToRelConverter.java:1762)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2
[jira] [Commented] (DRILL-3344) When Group By clause is present, the argument in window function should not refer to any column outside Group By
[
https://issues.apache.org/jira/browse/DRILL-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606587#comment-14606587
]
Victoria Markman commented on DRILL-3344:
-
Verified fixed in 1.1
#Mon Jun 29 17:55:24 EDT 2015
git.commit.id.abbrev=e923ac5
{code}
0: jdbc:drill:schema=dfs> create table tblForView(col_int, col_bigint,
col_char_2, col_vchar_52, col_tmstmp, col_dt, col_booln, col_dbl, col_tm) as
select cast(columns[0] as INT), cast(columns[1] as BIGINT),cast(columns[2] as
CHAR(2)), cast(columns[3] as VARCHAR(52)), cast(columns[4] as TIMESTAMP),
cast(columns[5] as DATE), cast(columns[6] as BOOLEAN),cast(columns[7] as
DOUBLE),cast(columns[8] as TIME) from `forPrqView.csv`;
+---++
| Fragment | Number of records written |
+---++
| 0_0 | 30 |
+---++
1 row selected (2.904 seconds)
0: jdbc:drill:schema=dfs> select max(col_tm) over(), col_char_2 from
tblForView group by col_char_2;
Error: PARSE ERROR: From line 1, column 12 to line 1, column 17: Expression
'col_tm' is not being grouped
[Error Id: 5e49ce6c-c99b-4055-acb9-0946d63bd794 on atsqa4-133.qa.lab:31010]
(state=,code=0)
{code}
> When Group By clause is present, the argument in window function should not
> refer to any column outside Group By
>
>
> Key: DRILL-3344
> URL: https://issues.apache.org/jira/browse/DRILL-3344
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.1.0
> Environment: 6ebfbb9d0fc0b87b032f5e5d5cb0825f5464426e
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: no_verified_test, window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3344.1.patch, forPrqView.csv
>
>
> CTAS
> {code}
> 0: jdbc:drill:schema=dfs.tmp> create table tblForView(col_int, col_bigint,
> col_char_2, col_vchar_52, col_tmstmp, col_dt, col_booln, col_dbl, col_tm) as
> select cast(columns[0] as INT), cast(columns[1] as BIGINT),cast(columns[2] as
> CHAR(2)), cast(columns[3] as VARCHAR(52)), cast(columns[4] as TIMESTAMP),
> cast(columns[5] as DATE), cast(columns[6] as BOOLEAN),cast(columns[7] as
> DOUBLE),cast(columns[8] as TIME) from `forPrqView.csv`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 30 |
> +---++
> 1 row selected (0.586 seconds)
> {code}
> Failing query
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select max(col_tm) over(), col_char_2 from
> tblForView group by col_char_2;
> Error: SYSTEM ERROR: java.lang.AssertionError: Internal error: while
> converting MAX(`tblForView`.`col_tm`)
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace
> {code}
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> java.lang.AssertionError: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
> ~[drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:840)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:782)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:784)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:893)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
[jira] [Closed] (DRILL-3344) When Group By clause is present, the argument in window function should not refer to any column outside Group By
[
https://issues.apache.org/jira/browse/DRILL-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3344.
---
> When Group By clause is present, the argument in window function should not
> refer to any column outside Group By
>
>
> Key: DRILL-3344
> URL: https://issues.apache.org/jira/browse/DRILL-3344
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.1.0
> Environment: 6ebfbb9d0fc0b87b032f5e5d5cb0825f5464426e
>Reporter: Khurram Faraaz
>Assignee: Sean Hsuan-Yi Chu
> Labels: no_verified_test, window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3344.1.patch, forPrqView.csv
>
>
> CTAS
> {code}
> 0: jdbc:drill:schema=dfs.tmp> create table tblForView(col_int, col_bigint,
> col_char_2, col_vchar_52, col_tmstmp, col_dt, col_booln, col_dbl, col_tm) as
> select cast(columns[0] as INT), cast(columns[1] as BIGINT),cast(columns[2] as
> CHAR(2)), cast(columns[3] as VARCHAR(52)), cast(columns[4] as TIMESTAMP),
> cast(columns[5] as DATE), cast(columns[6] as BOOLEAN),cast(columns[7] as
> DOUBLE),cast(columns[8] as TIME) from `forPrqView.csv`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 30 |
> +---++
> 1 row selected (0.586 seconds)
> {code}
> Failing query
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select max(col_tm) over(), col_char_2 from
> tblForView group by col_char_2;
> Error: SYSTEM ERROR: java.lang.AssertionError: Internal error: while
> converting MAX(`tblForView`.`col_tm`)
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace
> {code}
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> java.lang.AssertionError: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> [Error Id: 11afbdc9-d47a-4a52-aa77-40c20ffd2bc6 on centos-03.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
> ~[drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:738)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:840)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:782)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:73)
> [drill-common-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman$StateSwitch.moveToState(Foreman.java:784)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:893)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:253)
> [drill-java-exec-1.1.0-SNAPSHOT-rebuffed.jar:1.1.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: org.apache.drill.exec.work.foreman.ForemanException: Unexpected
> exception during fragment initialization: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> ... 4 common frames omitted
> Caused by: java.lang.AssertionError: Internal error: while converting
> MAX(`tblForView`.`col_tm`)
> at org.apache.calcite.util.Util.newInternal(Util.java:790)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.ReflectiveConvertletTable$2.convertCall(ReflectiveConvertletTable.java:152)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertOver(SqlToRelConverter.java:1762)
> ~[calcite-core-1.1.0-drill-r8.jar:1.1.0-drill-r8]
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.access$1000(SqlToRelConverter.java:180)
> ~[calc
[jira] [Closed] (DRILL-3220) IOB Exception when using constants in window functions
[
https://issues.apache.org/jira/browse/DRILL-3220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3220.
---
> IOB Exception when using constants in window functions
> --
>
> Key: DRILL-3220
> URL: https://issues.apache.org/jira/browse/DRILL-3220
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.0.0
> Environment: faec150598840c40827e6493992d81209aa936da
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3220.1.patch.txt, DRILL-3220.2.patch.txt,
> DRILL-3220.3.patch.txt
>
>
> The following query should run, but it throws an IndexOutOfBoundsException
> instead:
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select count(1) over(partition by columns[1]
> order by columns[0]) from `time_data.csv`;
> Error: SYSTEM ERROR: java.lang.IndexOutOfBoundsException: index (2) must be
> less than size (2)
> [Error Id: ddcd3316-e906-4193-8e55-e208c3cda32d on centos-03.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: index (2) must be less than size (2)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:251)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: java.lang.IndexOutOfBoundsException: index (2) must be less than
> size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> ~[guava-14.0.1.jar:na]
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> ~[guava-14.0.1.jar:na]
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> ~[guava-14.0.1.jar:na]
> at org.apache.calcite.util.Pair$6.get(Pair.java:335)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.drill.exec.planner.physical.WindowPrel.toDrill(WindowPrel.java:112)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.WindowPrel.getPhysicalOperator(WindowPrel.java:89)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.ProjectPrel.getPhysicalOperator(ProjectPrel.java:55)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.ProjectPrel.getPhysicalOperator(ProjectPrel.java:55)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.ScreenPrel.getPhysicalOperator(ScreenPrel.java:51)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPop(DefaultSqlHandler.java:447)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:192)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:177)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:902)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:240)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> ... 3 common frames omitted
> 2015-05-30 01:22:07,372 [2a96ef40-25ae-aedc-9f56-2da309e4d911:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up.
> 2015-05-30 01:22:07,373 [2a96ef40-25ae-aedc-9f56-2da309e4d911:foreman] ERROR
> o.a.d.c.exceptions.UserException - SYSTEM ERROR:
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> [Error Id: ddcd3316-e906-4193-8e55-e208c3cda32d on centos-03.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> [Error Id: ddcd3316-e906-4193-8e55-e208c3cda32d on centos-03.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
> ~[drill-common-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apach
[jira] [Commented] (DRILL-3220) IOB Exception when using constants in window functions
[
https://issues.apache.org/jira/browse/DRILL-3220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606597#comment-14606597
]
Victoria Markman commented on DRILL-3220:
-
Verified fixed in 1.1
#Mon Jun 29 17:55:24 EDT 2015
git.commit.id.abbrev=e923ac5
Test : Functional/Passing/window_functions/bugs/drill-3220.sql
> IOB Exception when using constants in window functions
> --
>
> Key: DRILL-3220
> URL: https://issues.apache.org/jira/browse/DRILL-3220
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.0.0
> Environment: faec150598840c40827e6493992d81209aa936da
>Reporter: Khurram Faraaz
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3220.1.patch.txt, DRILL-3220.2.patch.txt,
> DRILL-3220.3.patch.txt
>
>
> The following query should run, but it throws an IndexOutOfBoundsException
> instead:
> {code}
> 0: jdbc:drill:schema=dfs.tmp> select count(1) over(partition by columns[1]
> order by columns[0]) from `time_data.csv`;
> Error: SYSTEM ERROR: java.lang.IndexOutOfBoundsException: index (2) must be
> less than size (2)
> [Error Id: ddcd3316-e906-4193-8e55-e208c3cda32d on centos-03.qa.lab:31010]
> (state=,code=0)
> {code}
> Stack trace from drillbit.log
> {code}
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: index (2) must be less than size (2)
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:251)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Caused by: java.lang.IndexOutOfBoundsException: index (2) must be less than
> size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> ~[guava-14.0.1.jar:na]
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> ~[guava-14.0.1.jar:na]
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> ~[guava-14.0.1.jar:na]
> at org.apache.calcite.util.Pair$6.get(Pair.java:335)
> ~[calcite-core-1.1.0-drill-r7.jar:1.1.0-drill-r7]
> at
> org.apache.drill.exec.planner.physical.WindowPrel.toDrill(WindowPrel.java:112)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.WindowPrel.getPhysicalOperator(WindowPrel.java:89)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.ProjectPrel.getPhysicalOperator(ProjectPrel.java:55)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.ProjectPrel.getPhysicalOperator(ProjectPrel.java:55)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.physical.ScreenPrel.getPhysicalOperator(ScreenPrel.java:51)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToPop(DefaultSqlHandler.java:447)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:192)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:177)
> ~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:902)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:240)
> [drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
> ... 3 common frames omitted
> 2015-05-30 01:22:07,372 [2a96ef40-25ae-aedc-9f56-2da309e4d911:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - foreman cleaning up.
> 2015-05-30 01:22:07,373 [2a96ef40-25ae-aedc-9f56-2da309e4d911:foreman] ERROR
> o.a.d.c.exceptions.UserException - SYSTEM ERROR:
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> [Error Id: ddcd3316-e906-4193-8e55-e208c3cda32d on centos-03.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> [Error Id: ddcd3316-e906-4193-8e55-e208c3cda32d on centos
[jira] [Commented] (DRILL-3254) Average over window functions returns wrong results
[
https://issues.apache.org/jira/browse/DRILL-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606613#comment-14606613
]
Victoria Markman commented on DRILL-3254:
-
Verified fixed in 1.1
#Mon Jun 29 17:55:24 EDT 2015
git.commit.id.abbrev=e923ac5
Tests in Functional/Passing/window_functions/misc
> Average over window functions returns wrong results
> ---
>
> Key: DRILL-3254
> URL: https://issues.apache.org/jira/browse/DRILL-3254
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Abhishek Girish
>Assignee: Mehant Baid
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3254.patch
>
>
> Average function on numeric column returns an (inaccurate) integer value,
> instead of an (accurate) decimal (or floating point) value.
> *Results from Drill:*
> {code:sql}
> > select s_city, s_store_sk, avg(s_number_employees) over (PARTITION BY
> > s_city ORDER BY s_store_sk) from store limit 10;
> +---+-+-+
> | s_city | s_store_sk | EXPR$2 |
> +---+-+-+
> | Fairview | 5 | 288 |
> | Fairview | 8 | 283 |
> | Fairview | 12 | 286 |
> | Midway| 1 | 245 |
> | Midway| 2 | 240 |
> | Midway| 3 | 239 |
> | Midway| 4 | 233 |
> | Midway| 6 | 232 |
> | Midway| 7 | 243 |
> | Midway| 9 | 247 |
> +---+-+-+
> 10 rows selected (0.197 seconds)
> {code}
> *Results from Postgres:*
> {code:sql}
> # select s_city, s_store_sk, avg(s_number_employees) over (PARTITION BY
> s_city ORDER BY s_store_sk) from store limit 10;
> s_city | s_store_sk | avg
> --++--
> Fairview | 5 | 288.
> Fairview | 8 | 283.
> Fairview | 12 | 286.6667
> Midway | 1 | 245.
> Midway | 2 | 240.5000
> Midway | 3 | 239.
> Midway | 4 | 233.7500
> Midway | 6 | 232.8000
> Midway | 7 | 243.5000
> Midway | 9 | 247.4285714285714286
> (10 rows)
> {code}
> Drill returns right results without window functions:
> {code:sql}
> > select s_city, s_store_sk, avg(s_number_employees) from store group by
> > s_city, s_store_sk order by 1,2 limit 10;
> +---+-+-+
> | s_city | s_store_sk | EXPR$2 |
> +---+-+-+
> | Fairview | 5 | 288.0 |
> | Fairview | 8 | 278.0 |
> | Fairview | 12 | 294.0 |
> | Midway| 1 | 245.0 |
> | Midway| 2 | 236.0 |
> | Midway| 3 | 236.0 |
> | Midway| 4 | 218.0 |
> | Midway| 6 | 229.0 |
> | Midway| 7 | 297.0 |
> | Midway| 9 | 271.0 |
> +---+-+-+
> 10 rows selected (0.306 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3254) Average over window functions returns wrong results
[
https://issues.apache.org/jira/browse/DRILL-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3254.
---
> Average over window functions returns wrong results
> ---
>
> Key: DRILL-3254
> URL: https://issues.apache.org/jira/browse/DRILL-3254
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Abhishek Girish
>Assignee: Mehant Baid
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3254.patch
>
>
> Average function on numeric column returns an (inaccurate) integer value,
> instead of an (accurate) decimal (or floating point) value.
> *Results from Drill:*
> {code:sql}
> > select s_city, s_store_sk, avg(s_number_employees) over (PARTITION BY
> > s_city ORDER BY s_store_sk) from store limit 10;
> +---+-+-+
> | s_city | s_store_sk | EXPR$2 |
> +---+-+-+
> | Fairview | 5 | 288 |
> | Fairview | 8 | 283 |
> | Fairview | 12 | 286 |
> | Midway| 1 | 245 |
> | Midway| 2 | 240 |
> | Midway| 3 | 239 |
> | Midway| 4 | 233 |
> | Midway| 6 | 232 |
> | Midway| 7 | 243 |
> | Midway| 9 | 247 |
> +---+-+-+
> 10 rows selected (0.197 seconds)
> {code}
> *Results from Postgres:*
> {code:sql}
> # select s_city, s_store_sk, avg(s_number_employees) over (PARTITION BY
> s_city ORDER BY s_store_sk) from store limit 10;
> s_city | s_store_sk | avg
> --++--
> Fairview | 5 | 288.
> Fairview | 8 | 283.
> Fairview | 12 | 286.6667
> Midway | 1 | 245.
> Midway | 2 | 240.5000
> Midway | 3 | 239.
> Midway | 4 | 233.7500
> Midway | 6 | 232.8000
> Midway | 7 | 243.5000
> Midway | 9 | 247.4285714285714286
> (10 rows)
> {code}
> Drill returns right results without window functions:
> {code:sql}
> > select s_city, s_store_sk, avg(s_number_employees) from store group by
> > s_city, s_store_sk order by 1,2 limit 10;
> +---+-+-+
> | s_city | s_store_sk | EXPR$2 |
> +---+-+-+
> | Fairview | 5 | 288.0 |
> | Fairview | 8 | 278.0 |
> | Fairview | 12 | 294.0 |
> | Midway| 1 | 245.0 |
> | Midway| 2 | 236.0 |
> | Midway| 3 | 236.0 |
> | Midway| 4 | 218.0 |
> | Midway| 6 | 229.0 |
> | Midway| 7 | 297.0 |
> | Midway| 9 | 271.0 |
> +---+-+-+
> 10 rows selected (0.306 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3188) Restrict the types of window frames that can be specified
[
https://issues.apache.org/jira/browse/DRILL-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606621#comment-14606621
]
Victoria Markman commented on DRILL-3188:
-
Verified fixed in 1.1
#Mon Jun 29 17:55:24 EDT 2015
git.commit.id.abbrev=e923ac5
Following bugs were opened as a follow up:
DRILL-3325 - Explicitly specified default window frame throws an error
requiring order by
DRILL-3409 - Specifying default frame explicitly results in an error
DRILL-3359 - Drill should throw and error when window function defined using
WINDOW AS uses ROWS UNBOUNDED PRECEDING
> Restrict the types of window frames that can be specified
> -
>
> Key: DRILL-3188
> URL: https://issues.apache.org/jira/browse/DRILL-3188
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.1.0
>
>
> We don't support row range with window functions. So we should disable this
> functionality, because currently we return default frame result.
> The only frame we currently support is BETWEEN UNBOUNDED PRECEDING AND
> CURRENT ROW.
> If you don't specify frame in Calcite, this is exactly what you get:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select a2,b2,c2, cast(count(*)
> over(partition by a2, substr(b2,1,2),c2 order by cast(a2 as double) + 100 )
> as bigint) from t2 order by a2;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(a2=[$0], b2=[$1], c2=[$2], EXPR$3=[$3])
> 00-02SelectionVectorRemover
> 00-03 Sort(sort0=[$0], dir0=[ASC])
> 00-04Project(a2=[$0], b2=[$1], c2=[$2], EXPR$3=[$5])
> 00-05 Window(window#0=[window(partition {0, 2, 4} order by [3]
> range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT()])])
> 00-06SelectionVectorRemover
> 00-07 Sort(sort0=[$0], sort1=[$2], sort2=[$4], sort3=[$3],
> dir0=[ASC], dir1=[ASC], dir2=[ASC], dir3=[ASC])
> 00-08Project(a2=[$1], b2=[$0], c2=[$2],
> $3=[+(CAST($1):DOUBLE, 100)], $4=[SUBSTR($0, 1, 2)])
> 00-09 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/aggregation/t2]],
> selectionRoot=/drill/testdata/aggregation/t2, numFiles=1, columns=[`a2`,
> `b2`, `c2`]]])
> {code}
> Row range should be disabled as well:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2;
> +-++-+
> | a2 | b2 | c2 |
> +-++-+
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> +-++-+
> 13 rows selected (0.123 seconds)
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following ) from t2 order by a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | 0 |
> | 1 | 1 |
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | 3 |
> | 4 | 4 |
> | 5 | 5 |
> | 6 | 6 |
> | 7 | 14 |
> | 7 | 14 |
> | 8 | 8 |
> | 9 | 9 |
> +-+-+
> 13 rows selected (0.2 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3188) Restrict the types of window frames that can be specified
[
https://issues.apache.org/jira/browse/DRILL-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3188.
---
> Restrict the types of window frames that can be specified
> -
>
> Key: DRILL-3188
> URL: https://issues.apache.org/jira/browse/DRILL-3188
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.1.0
>
>
> We don't support row range with window functions. So we should disable this
> functionality, because currently we return default frame result.
> The only frame we currently support is BETWEEN UNBOUNDED PRECEDING AND
> CURRENT ROW.
> If you don't specify frame in Calcite, this is exactly what you get:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select a2,b2,c2, cast(count(*)
> over(partition by a2, substr(b2,1,2),c2 order by cast(a2 as double) + 100 )
> as bigint) from t2 order by a2;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(a2=[$0], b2=[$1], c2=[$2], EXPR$3=[$3])
> 00-02SelectionVectorRemover
> 00-03 Sort(sort0=[$0], dir0=[ASC])
> 00-04Project(a2=[$0], b2=[$1], c2=[$2], EXPR$3=[$5])
> 00-05 Window(window#0=[window(partition {0, 2, 4} order by [3]
> range between UNBOUNDED PRECEDING and CURRENT ROW aggs [COUNT()])])
> 00-06SelectionVectorRemover
> 00-07 Sort(sort0=[$0], sort1=[$2], sort2=[$4], sort3=[$3],
> dir0=[ASC], dir1=[ASC], dir2=[ASC], dir3=[ASC])
> 00-08Project(a2=[$1], b2=[$0], c2=[$2],
> $3=[+(CAST($1):DOUBLE, 100)], $4=[SUBSTR($0, 1, 2)])
> 00-09 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/aggregation/t2]],
> selectionRoot=/drill/testdata/aggregation/t2, numFiles=1, columns=[`a2`,
> `b2`, `c2`]]])
> {code}
> Row range should be disabled as well:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2;
> +-++-+
> | a2 | b2 | c2 |
> +-++-+
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> +-++-+
> 13 rows selected (0.123 seconds)
> 0: jdbc:drill:schema=dfs> select a2, sum(a2) over(partition by a2 order by a2
> rows between 1 preceding and 1 following ) from t2 order by a2;
> +-+-+
> | a2 | EXPR$1 |
> +-+-+
> | 0 | 0 |
> | 1 | 1 |
> | 2 | 6 |
> | 2 | 6 |
> | 2 | 6 |
> | 3 | 3 |
> | 4 | 4 |
> | 5 | 5 |
> | 6 | 6 |
> | 7 | 14 |
> | 7 | 14 |
> | 8 | 8 |
> | 9 | 9 |
> +-+-+
> 13 rows selected (0.2 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3358) CUME_DIST window function provides wrong result when only ORDER BY clause is specified
[
https://issues.apache.org/jira/browse/DRILL-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606631#comment-14606631
]
Victoria Markman commented on DRILL-3358:
-
I get correct result with 1.1
#Mon Jun 29 17:55:24 EDT 2015
git.commit.id.abbrev=e923ac5
{code}
0: jdbc:drill:schema=dfs> select rank() over w rnk, percent_rank() over w
prct_rnk, dense_rank() over w denseRank, row_number() over w row_num,
cume_dist() over w cumedist, col_char_2 from tblForView window w as (partition
by col_char_2 order by col_int) order by 1;
+--+---++--+-+-+
| rnk | prct_rnk | denseRank | row_num | cumedist | col_char_2 |
+--+---++--+-+-+
| 1| 0.0 | 1 | 1| 1.0 | AZ |
| 1| 0.0 | 1 | 1| 0.5 | CO |
| 1| 0.0 | 1 | 1| 1.0 | FL |
| 1| 0.0 | 1 | 1| 0.5 | GA |
| 1| 0.0 | 1 | 1| 1.0 | HI |
| 1| 0.0 | 1 | 1| 0.5 | IN |
| 1| 0.0 | 1 | 1| 0. | MA |
| 1| 0.0 | 1 | 1| 0.5 | MD |
| 1| 0.0 | 1 | 1| 1.0 | MI |
| 1| 0.0 | 1 | 1| 0.5 | MN |
| 1| 0.0 | 1 | 1| 1.0 | MO |
| 1| 0.0 | 1 | 1| 0.5 | ND |
| 1| 0.0 | 1 | 1| 1.0 | OR |
| 1| 0.0 | 1 | 1| 0. | RI |
| 1| 0.0 | 1 | 1| 0.5 | SD |
| 1| 0.0 | 1 | 1| 0.5 | VT |
| 1| 0.0 | 1 | 1| 1.0 | WI |
| 1| 0.0 | 1 | 1| 1.0 | WY |
| 2| 1.0 | 2 | 2| 1.0 | MN |
| 2| 0.5 | 2 | 2| 0. | RI |
| 2| 1.0 | 2 | 2| 1.0 | IN |
| 2| 0.5 | 2 | 2| 0. | MA |
| 2| 1.0 | 2 | 2| 1.0 | ND |
| 2| 1.0 | 2 | 2| 1.0 | MD |
| 2| 1.0 | 2 | 2| 1.0 | CO |
| 2| 1.0 | 2 | 2| 1.0 | VT |
| 2| 1.0 | 2 | 2| 1.0 | SD |
| 2| 1.0 | 2 | 2| 1.0 | GA |
| 3| 1.0 | 3 | 3| 1.0 | RI |
| 3| 1.0 | 3 | 3| 1.0 | MA |
+--+---++--+-+-+
30 rows selected (0.486 seconds)
{code}
> CUME_DIST window function provides wrong result when only ORDER BY clause is
> specified
> --
>
> Key: DRILL-3358
> URL: https://issues.apache.org/jira/browse/DRILL-3358
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Abhishek Girish
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.1.0
>
>
> *Drill:*
> {code:sql}
> > SELECT CUME_DIST() OVER (ORDER BY ss.ss_store_sk) FROM store_sales ss ORDER
> > BY 1 LIMIT 20;
> +-+
> | EXPR$0|
> +-+
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> +-+
> 20 rows selected (17.317 seconds)
> {code}
> *Postgres*
> {code:sql}
> # SELECT CUME_DIST() OVER (ORDER BY ss.ss_store_sk) FROM store_sales ss ORDER
> BY 1 LIMIT 20;
> cume_dist
> ---
> 0.158622193275665
> 0.1586
[jira] [Closed] (DRILL-3358) CUME_DIST window function provides wrong result when only ORDER BY clause is specified
[
https://issues.apache.org/jira/browse/DRILL-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3358.
---
> CUME_DIST window function provides wrong result when only ORDER BY clause is
> specified
> --
>
> Key: DRILL-3358
> URL: https://issues.apache.org/jira/browse/DRILL-3358
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.1.0
>Reporter: Abhishek Girish
>Assignee: Deneche A. Hakim
> Labels: window_function
> Fix For: 1.1.0
>
>
> *Drill:*
> {code:sql}
> > SELECT CUME_DIST() OVER (ORDER BY ss.ss_store_sk) FROM store_sales ss ORDER
> > BY 1 LIMIT 20;
> +-+
> | EXPR$0|
> +-+
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> | 0.9923989432198661 |
> +-+
> 20 rows selected (17.317 seconds)
> {code}
> *Postgres*
> {code:sql}
> # SELECT CUME_DIST() OVER (ORDER BY ss.ss_store_sk) FROM store_sales ss ORDER
> BY 1 LIMIT 20;
> cume_dist
> ---
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> 0.158622193275665
> (20 rows)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3091) Cancelled query continues to list on Drill UI with CANCELLATION_REQUESTED state
[ https://issues.apache.org/jira/browse/DRILL-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609221#comment-14609221 ] Victoria Markman commented on DRILL-3091: - I'm observing the same in 1.1 ... we need to get to the bottom of this soon. > Cancelled query continues to list on Drill UI with CANCELLATION_REQUESTED > state > --- > > Key: DRILL-3091 > URL: https://issues.apache.org/jira/browse/DRILL-3091 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 1.0.0 >Reporter: Abhishek Girish >Assignee: Chris Westin > Attachments: drillbit.log > > > A long running query (TPC-DS SF 100 - query 2) continues to be listed on the > Drill UI query profile page, among the list of running queries. It's been > more than 30 minutes as of this report. > TOP -p showed no activity after the cancellation. And > Jstack on all nodes did not contain the queryID. > I can share more details for repro. > Git.Commit.ID: 583ca4a (May 14 build) > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3091) Cancelled query continues to list on Drill UI with CANCELLATION_REQUESTED state
[ https://issues.apache.org/jira/browse/DRILL-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-3091: Fix Version/s: 1.2.0 > Cancelled query continues to list on Drill UI with CANCELLATION_REQUESTED > state > --- > > Key: DRILL-3091 > URL: https://issues.apache.org/jira/browse/DRILL-3091 > Project: Apache Drill > Issue Type: Bug > Components: Client - HTTP >Affects Versions: 1.0.0 >Reporter: Abhishek Girish >Assignee: Chris Westin > Fix For: 1.2.0 > > Attachments: drillbit.log > > > A long running query (TPC-DS SF 100 - query 2) continues to be listed on the > Drill UI query profile page, among the list of running queries. It's been > more than 30 minutes as of this report. > TOP -p showed no activity after the cancellation. And > Jstack on all nodes did not contain the queryID. > I can share more details for repro. > Git.Commit.ID: 583ca4a (May 14 build) > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3292) SUM(constant) OVER(...) returns wrong results
[
https://issues.apache.org/jira/browse/DRILL-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3292:
Labels: window_function (was: )
> SUM(constant) OVER(...) returns wrong results
> -
>
> Key: DRILL-3292
> URL: https://issues.apache.org/jira/browse/DRILL-3292
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators, Query Planning &
> Optimization
>Affects Versions: 1.0.0
>Reporter: Deneche A. Hakim
>Assignee: Sean Hsuan-Yi Chu
>Priority: Critical
> Labels: window_function
> Fix For: 1.2.0
>
>
> The following query returns wrong results:
> {noformat}
> 0: jdbc:drill:> select sum(1) over w sum1, sum(5) over w sum5 from
> cp.`employee.json` where position_id = 2 window w as (partition by
> position_id);
> +---+---+
> | sum1 | sum5 |
> +---+---+
> | 6 | 6 |
> | 6 | 6 |
> | 6 | 6 |
> | 6 | 6 |
> | 6 | 6 |
> | 6 | 6 |
> +---+---+
> {noformat}
> The second column should display 30 (5 x 6) instead of 6.
> Here is the plan for the query:
> {noformat}
> 00-00Screen
> 00-01 Project(sum1=[$0], sum5=[$1])
> 00-02Project(sum1=[$0], sum5=[$1])
> 00-03 Project($0=[$1], $1=[$2])
> 00-04Window(window#0=[window(partition {0} order by [] range
> between UNBOUNDED PRECEDING and UNBOUNDED FOLLOWING aggs [SUM($1), SUM($2)])])
> 00-05 SelectionVectorRemover
> 00-06Sort(sort0=[$0], dir0=[ASC])
> 00-07 Filter(condition=[=($0, 2)])
> 00-08Scan(groupscan=[EasyGroupScan
> [selectionRoot=/employee.json, numFiles=1, columns=[`position_id`],
> files=[classpath:/employee.json]]])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3268) queries with empty OVER() clause return empty result set
[
https://issues.apache.org/jira/browse/DRILL-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-3268.
---
> queries with empty OVER() clause return empty result set
>
>
> Key: DRILL-3268
> URL: https://issues.apache.org/jira/browse/DRILL-3268
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Steven Phillips
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3268.1.patch.txt
>
>
> The following query returns, incorrectly, an empty result set:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT employee_id, position_id, salary, SUM(salary)
> OVER() FROM cp.`employee.json` LIMIT 1000;
> +--+--+-+-+
> | employee_id | position_id | salary | EXPR$3 |
> +--+--+-+-+
> +--+--+-+-+
> No rows selected (0.287 seconds)
> {noformat}
> DRILL-1862 is required otherwise such queries won't plan at all.
> The problem is caused by WindowFrameRecordBatch.buildSchema() which assumes
> the very first batch to be an empty batch that only contains the schema,
> which is not the case when using OVER()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3268) queries with empty OVER() clause return empty result set
[
https://issues.apache.org/jira/browse/DRILL-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612328#comment-14612328
]
Victoria Markman commented on DRILL-3268:
-
Verified fixed in 1.1
#Tue Jun 30 14:50:20 EDT 2015
git.commit.id.abbrev=e3fc7e9
Tests are in: Functional/Passing/window_functions/empty_over_clause
> queries with empty OVER() clause return empty result set
>
>
> Key: DRILL-3268
> URL: https://issues.apache.org/jira/browse/DRILL-3268
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Deneche A. Hakim
>Assignee: Steven Phillips
> Labels: window_function
> Fix For: 1.1.0
>
> Attachments: DRILL-3268.1.patch.txt
>
>
> The following query returns, incorrectly, an empty result set:
> {noformat}
> 0: jdbc:drill:zk=local> SELECT employee_id, position_id, salary, SUM(salary)
> OVER() FROM cp.`employee.json` LIMIT 1000;
> +--+--+-+-+
> | employee_id | position_id | salary | EXPR$3 |
> +--+--+-+-+
> +--+--+-+-+
> No rows selected (0.287 seconds)
> {noformat}
> DRILL-1862 is required otherwise such queries won't plan at all.
> The problem is caused by WindowFrameRecordBatch.buildSchema() which assumes
> the very first batch to be an empty batch that only contains the schema,
> which is not the case when using OVER()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3297) Using rank, dense_rank, percent_rank, cume_dist, row_number window functions without OVER clause results in cryptic schema change error
[
https://issues.apache.org/jira/browse/DRILL-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3297:
Labels: window_function (was: )
> Using rank, dense_rank, percent_rank, cume_dist, row_number window functions
> without OVER clause results in cryptic schema change error
> ---
>
> Key: DRILL-3297
> URL: https://issues.apache.org/jira/browse/DRILL-3297
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Labels: window_function
> Fix For: 1.3.0
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select rank() from t1;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> org.apache.drill.exec.exception.SchemaChangeException: Failure while
> materializing expression.
> Error in expression at index -1. Error: Missing function implementation:
> [rank(BIGINT-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.
> Fragment 0:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3029) Wrong result with correlated not exists subquery
[
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3029:
Attachment: t1_t2_t3.tar
> Wrong result with correlated not exists subquery
>
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 9 |
> | 10 |
> | null |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3029) Wrong result with correlated not exists subquery
[
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612484#comment-14612484
]
Victoria Markman commented on DRILL-3029:
-
Drill output with 1.1
#Tue Jun 30 14:50:20 EDT 2015
git.commit.id.abbrev=e3fc7e9
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > distinct a1
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t1
. . . . . . . . . . . . > where not exists
. . . . . . . . . . . . > (
. . . . . . . . . . . . > select
. . . . . . . . . . . . > *
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t2
. . . . . . . . . . . . > where not exists
. . . . . . . . . . . . > (
. . . . . . . . . . . . > select
. . . . . . . . . . . . > *
. . . . . . . . . . . . > from
. . . . . . . . . . . . > t3
. . . . . . . . . . . . > where
. . . . . . . . . . . . > t3.b3 = t2.b2 and
. . . . . . . . . . . . > t3.a3 = t1.a1
. . . . . . . . . . . . > )
. . . . . . . . . . . . > )
. . . . . . . . . . . . > ;
+---+
| a1 |
+---+
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
| 7 |
| 9 |
| 10|
| null |
+---+
10 rows selected (3.633 seconds)
{code}
> Wrong result with correlated not exists subquery
>
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 9 |
> | 10 |
> | null |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3029) Wrong result with correlated not exists subquery
[
https://issues.apache.org/jira/browse/DRILL-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3029:
Assignee: Jinfeng Ni (was: Victoria Markman)
> Wrong result with correlated not exists subquery
>
>
> Key: DRILL-3029
> URL: https://issues.apache.org/jira/browse/DRILL-3029
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 1.0.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
>Priority: Critical
> Fix For: 1.2.0
>
> Attachments: t1_t2_t3.tar
>
>
> Subquery has correlation to two outer tables in the previous blocks.
> Postgres returns empty result set in this case:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > distinct a1
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t1
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t2
> . . . . . . . . . . . . > where not exists
> . . . . . . . . . . . . > (
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > *
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > t3
> . . . . . . . . . . . . > where
> . . . . . . . . . . . . > t3.b3 = t2.b2 and
> . . . . . . . . . . . . > t3.a3 = t1.a1
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > )
> . . . . . . . . . . . . > ;
> ++
> | a1 |
> ++
> | 1 |
> | 2 |
> | 3 |
> | 4 |
> | 5 |
> | 6 |
> | 7 |
> | 9 |
> | 10 |
> | null |
> ++
> 10 rows selected (0.991 seconds)
> {code}
> Copy/paste reproduction:
> {code}
> select
> distinct a1
> from
> t1
> where not exists
> (
> select
> *
> from
> t2
> where not exists
> (
> select
> *
> from
> t3
> where
> t3.b3 = t2.b2 and
> t3.a3 = t1.a1
> )
> )
> ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2735) Broadcast plan get's "lost" when the same query is used in UNION ALL
[
https://issues.apache.org/jira/browse/DRILL-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2735:
Attachment: j1_j2_tables.tar
> Broadcast plan get's "lost" when the same query is used in UNION ALL
>
>
> Key: DRILL-2735
> URL: https://issues.apache.org/jira/browse/DRILL-2735
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: j1_j2_tables.tar
>
>
> I get a broadcast plan for simple inner join query.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
> where j1.c_integer = j2.c_integer;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 UnionExchange
> 01-01Project(c_integer=[$0])
> 01-02 HashJoin(condition=[=($0, $1)], joinType=[inner])
> 01-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/ctas/j1]],
> selectionRoot=/drill/testdata/ctas/j1, numFiles=1, columns=[`c_integer`]]])
> 01-03Project(c_integer0=[$0])
> 01-05 BroadcastExchange
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/ctas/j2]],
> selectionRoot=/drill/testdata/ctas/j2, numFiles=1, columns=[`c_integer`]]])
> | {
> "head" : {
> "version" : 1,
> "generator" : {
> "type" : "ExplainHandler",
> "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ {
> "name" : "planner.broadcast_factor",
> "kind" : "DOUBLE",
> "type" : "SESSION",
> "float_val" : 0.0
> }, {
> "name" : "planner.slice_target",
> "kind" : "LONG",
> "type" : "SESSION",
> "num_val" : 1
> } ],
> {code}
> Create table succeeds and multiple fragments are executed:
> {code}
> 0: jdbc:drill:schema=dfs> create table test(a1) as select j1.c_integer from
> j1, j2 where j1.c_integer = j2.c_integer;
> ++---+
> | Fragment | Number of records written |
> ++---+
> | 1_1| 0 |
> | 1_3| 0 |
> | 1_31 | 0 |
> | 1_43 | 0 |
> | 1_35 | 0 |
> | 1_21 | 0 |
> | 1_19 | 0 |
> | 1_27 | 1 |
> | 1_17 | 1 |
> | 1_13 | 0 |
> | 1_29 | 0 |
> | 1_33 | 0 |
> | 1_25 | 0 |
> | 1_7| 0 |
> | 1_11 | 0 |
> | 1_37 | 0 |
> | 1_45 | 0 |
> | 1_9| 0 |
> | 1_23 | 1 |
> | 1_15 | 0 |
> | 1_41 | 0 |
> | 1_39 | 0 |
> | 1_5| 0 |
> | 1_10 | 0 |
> | 1_14 | 0 |
> | 1_24 | 0 |
> | 1_16 | 0 |
> | 1_12 | 0 |
> | 1_36 | 0 |
> | 1_20 | 0 |
> | 1_34 | 1 |
> | 1_40 | 0 |
> | 1_22 | 0 |
> | 1_26 | 0 |
> | 1_32 | 1 |
> | 1_8| 0 |
> | 1_18 | 0 |
> | 1_42 | 0 |
> | 1_44 | 0 |
> | 1_38 | 0 |
> | 1_30 | 0 |
> | 1_28 | 1 |
> | 1_4| 10|
> | 1_2| 1 |
> | 1_6| 0 |
> | 1_0| 0 |
> ++---+
> 46 rows selected (2.337 seconds)
> {code}
> 8 parquet files are written:
> {code}
> [Wed Apr 08 11:41:10 root@/mapr/vmarkman.cluster.com/drill/testdata/ctas/test
> ] # ls -ltr
> total 4
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_17_0.parquet
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_27_0.parquet
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_23_0.parquet
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:
[jira] [Commented] (DRILL-2735) Broadcast plan get's "lost" when the same query is used in UNION ALL
[
https://issues.apache.org/jira/browse/DRILL-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612630#comment-14612630
]
Victoria Markman commented on DRILL-2735:
-
Still reproduces in 1.1:
{code}
0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
where j1.c_integer = j2.c_integer union all select j1.c_integer from j1, j2
where j1.c_integer = j2.c_integer;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 UnionAll(all=[true])
00-03Project(c_integer=[$0])
00-05 HashJoin(condition=[=($0, $1)], joinType=[inner])
00-09Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j1]],
selectionRoot=maprfs:/drill/testdata/subqueries/j1, numFiles=1,
columns=[`c_integer`]]])
00-08Project(c_integer0=[$0])
00-11 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j2]],
selectionRoot=maprfs:/drill/testdata/subqueries/j2, numFiles=1,
columns=[`c_integer`]]])
00-02Project(c_integer=[$0])
00-04 HashJoin(condition=[=($0, $1)], joinType=[inner])
00-07Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j1]],
selectionRoot=maprfs:/drill/testdata/subqueries/j1, numFiles=1,
columns=[`c_integer`]]])
00-06Project(c_integer0=[$0])
00-10 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j2]],
selectionRoot=maprfs:/drill/testdata/subqueries/j2, numFiles=1,
columns=[`c_integer`]]])
{code}
> Broadcast plan get's "lost" when the same query is used in UNION ALL
>
>
> Key: DRILL-2735
> URL: https://issues.apache.org/jira/browse/DRILL-2735
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: j1_j2_tables.tar
>
>
> I get a broadcast plan for simple inner join query.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
> where j1.c_integer = j2.c_integer;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 UnionExchange
> 01-01Project(c_integer=[$0])
> 01-02 HashJoin(condition=[=($0, $1)], joinType=[inner])
> 01-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/ctas/j1]],
> selectionRoot=/drill/testdata/ctas/j1, numFiles=1, columns=[`c_integer`]]])
> 01-03Project(c_integer0=[$0])
> 01-05 BroadcastExchange
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/ctas/j2]],
> selectionRoot=/drill/testdata/ctas/j2, numFiles=1, columns=[`c_integer`]]])
> | {
> "head" : {
> "version" : 1,
> "generator" : {
> "type" : "ExplainHandler",
> "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ {
> "name" : "planner.broadcast_factor",
> "kind" : "DOUBLE",
> "type" : "SESSION",
> "float_val" : 0.0
> }, {
> "name" : "planner.slice_target",
> "kind" : "LONG",
> "type" : "SESSION",
> "num_val" : 1
> } ],
> {code}
> Create table succeeds and multiple fragments are executed:
> {code}
> 0: jdbc:drill:schema=dfs> create table test(a1) as select j1.c_integer from
> j1, j2 where j1.c_integer = j2.c_integer;
> ++---+
> | Fragment | Number of records written |
> ++---+
> | 1_1| 0 |
> | 1_3| 0 |
> | 1_31 | 0 |
> | 1_43 | 0 |
> | 1_35 | 0 |
> | 1_21 | 0 |
> | 1_19 | 0 |
> | 1_27 | 1 |
> | 1_17 | 1 |
> | 1_13 | 0 |
> | 1_29 | 0 |
> | 1_33 | 0 |
> | 1_25 | 0 |
> | 1_7| 0 |
> | 1_11 | 0 |
> | 1_37 | 0 |
> | 1_45 | 0 |
> | 1_9| 0 |
> | 1_23 | 1 |
> | 1_15 | 0 |
> | 1_41 | 0 |
> | 1_39 | 0 |
> | 1_5
[jira] [Comment Edited] (DRILL-2735) Broadcast plan get's "lost" when the same query is used in UNION ALL
[
https://issues.apache.org/jira/browse/DRILL-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612630#comment-14612630
]
Victoria Markman edited comment on DRILL-2735 at 7/2/15 10:28 PM:
--
Still reproduces in 1.1 and produces single parquet file:
{code}
0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
where j1.c_integer = j2.c_integer union all select j1.c_integer from j1, j2
where j1.c_integer = j2.c_integer;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 UnionAll(all=[true])
00-03Project(c_integer=[$0])
00-05 HashJoin(condition=[=($0, $1)], joinType=[inner])
00-09Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j1]],
selectionRoot=maprfs:/drill/testdata/subqueries/j1, numFiles=1,
columns=[`c_integer`]]])
00-08Project(c_integer0=[$0])
00-11 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j2]],
selectionRoot=maprfs:/drill/testdata/subqueries/j2, numFiles=1,
columns=[`c_integer`]]])
00-02Project(c_integer=[$0])
00-04 HashJoin(condition=[=($0, $1)], joinType=[inner])
00-07Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j1]],
selectionRoot=maprfs:/drill/testdata/subqueries/j1, numFiles=1,
columns=[`c_integer`]]])
00-06Project(c_integer0=[$0])
00-10 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j2]],
selectionRoot=maprfs:/drill/testdata/subqueries/j2, numFiles=1,
columns=[`c_integer`]]])
{code}
was (Author: vicky):
Still reproduces in 1.1:
{code}
0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
where j1.c_integer = j2.c_integer union all select j1.c_integer from j1, j2
where j1.c_integer = j2.c_integer;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 UnionAll(all=[true])
00-03Project(c_integer=[$0])
00-05 HashJoin(condition=[=($0, $1)], joinType=[inner])
00-09Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j1]],
selectionRoot=maprfs:/drill/testdata/subqueries/j1, numFiles=1,
columns=[`c_integer`]]])
00-08Project(c_integer0=[$0])
00-11 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j2]],
selectionRoot=maprfs:/drill/testdata/subqueries/j2, numFiles=1,
columns=[`c_integer`]]])
00-02Project(c_integer=[$0])
00-04 HashJoin(condition=[=($0, $1)], joinType=[inner])
00-07Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j1]],
selectionRoot=maprfs:/drill/testdata/subqueries/j1, numFiles=1,
columns=[`c_integer`]]])
00-06Project(c_integer0=[$0])
00-10 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
[path=maprfs:///drill/testdata/subqueries/j2]],
selectionRoot=maprfs:/drill/testdata/subqueries/j2, numFiles=1,
columns=[`c_integer`]]])
{code}
> Broadcast plan get's "lost" when the same query is used in UNION ALL
>
>
> Key: DRILL-2735
> URL: https://issues.apache.org/jira/browse/DRILL-2735
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: j1_j2_tables.tar
>
>
> I get a broadcast plan for simple inner join query.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
> where j1.c_integer = j2.c_integer;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 UnionExchange
> 01-01Project(c_integer=[$0])
> 01-02 HashJoin(condition=[=($0, $1)], joinType=[inner])
> 01-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/ctas/j1]],
> selectionRoot=/drill/testdata/ctas/j1, numFiles=1, columns=[`c_integer`]]])
> 01-03Project(c_integer0=[$0])
> 01-05 BroadcastExchange
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/ctas/j2]],
> selectionRoot=/drill/testdata/ctas/j2, numFiles=1, columns=[`c_integer`]]])
> | {
> "head" : {
> "version" : 1,
> "generator" : {
> "type" : "ExplainHandler",
> "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ {
>
[jira] [Reopened] (DRILL-2735) Broadcast plan get's "lost" when the same query is used in UNION ALL
[
https://issues.apache.org/jira/browse/DRILL-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-2735:
-
> Broadcast plan get's "lost" when the same query is used in UNION ALL
>
>
> Key: DRILL-2735
> URL: https://issues.apache.org/jira/browse/DRILL-2735
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.2.0
>
> Attachments: j1_j2_tables.tar
>
>
> I get a broadcast plan for simple inner join query.
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select j1.c_integer from j1, j2
> where j1.c_integer = j2.c_integer;
> +++
> |text|json|
> +++
> | 00-00Screen
> 00-01 UnionExchange
> 01-01Project(c_integer=[$0])
> 01-02 HashJoin(condition=[=($0, $1)], joinType=[inner])
> 01-04Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=maprfs:/drill/testdata/ctas/j1]],
> selectionRoot=/drill/testdata/ctas/j1, numFiles=1, columns=[`c_integer`]]])
> 01-03Project(c_integer0=[$0])
> 01-05 BroadcastExchange
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/drill/testdata/ctas/j2]],
> selectionRoot=/drill/testdata/ctas/j2, numFiles=1, columns=[`c_integer`]]])
> | {
> "head" : {
> "version" : 1,
> "generator" : {
> "type" : "ExplainHandler",
> "info" : ""
> },
> "type" : "APACHE_DRILL_PHYSICAL",
> "options" : [ {
> "name" : "planner.broadcast_factor",
> "kind" : "DOUBLE",
> "type" : "SESSION",
> "float_val" : 0.0
> }, {
> "name" : "planner.slice_target",
> "kind" : "LONG",
> "type" : "SESSION",
> "num_val" : 1
> } ],
> {code}
> Create table succeeds and multiple fragments are executed:
> {code}
> 0: jdbc:drill:schema=dfs> create table test(a1) as select j1.c_integer from
> j1, j2 where j1.c_integer = j2.c_integer;
> ++---+
> | Fragment | Number of records written |
> ++---+
> | 1_1| 0 |
> | 1_3| 0 |
> | 1_31 | 0 |
> | 1_43 | 0 |
> | 1_35 | 0 |
> | 1_21 | 0 |
> | 1_19 | 0 |
> | 1_27 | 1 |
> | 1_17 | 1 |
> | 1_13 | 0 |
> | 1_29 | 0 |
> | 1_33 | 0 |
> | 1_25 | 0 |
> | 1_7| 0 |
> | 1_11 | 0 |
> | 1_37 | 0 |
> | 1_45 | 0 |
> | 1_9| 0 |
> | 1_23 | 1 |
> | 1_15 | 0 |
> | 1_41 | 0 |
> | 1_39 | 0 |
> | 1_5| 0 |
> | 1_10 | 0 |
> | 1_14 | 0 |
> | 1_24 | 0 |
> | 1_16 | 0 |
> | 1_12 | 0 |
> | 1_36 | 0 |
> | 1_20 | 0 |
> | 1_34 | 1 |
> | 1_40 | 0 |
> | 1_22 | 0 |
> | 1_26 | 0 |
> | 1_32 | 1 |
> | 1_8| 0 |
> | 1_18 | 0 |
> | 1_42 | 0 |
> | 1_44 | 0 |
> | 1_38 | 0 |
> | 1_30 | 0 |
> | 1_28 | 1 |
> | 1_4| 10|
> | 1_2| 1 |
> | 1_6| 0 |
> | 1_0| 0 |
> ++---+
> 46 rows selected (2.337 seconds)
> {code}
> 8 parquet files are written:
> {code}
> [Wed Apr 08 11:41:10 root@/mapr/vmarkman.cluster.com/drill/testdata/ctas/test
> ] # ls -ltr
> total 4
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_17_0.parquet
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_27_0.parquet
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_23_0.parquet
> -rwxr-xr-x 1 mapr mapr 146 Apr 8 11:40 1_34_0.parquet
> -rwxr-xr-x
[jira] [Commented] (DRILL-1611) The SQL fails Query failed: Failure while running fragment. Queue closed due to channel closure
[
https://issues.apache.org/jira/browse/DRILL-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615504#comment-14615504
]
Victoria Markman commented on DRILL-1611:
-
This query now runs successfully: version 1.1 of drill:
{code}
0: jdbc:drill:schema=dfs> SELECT A.c_customer_id,
. . . . . . . . . . . . > A.c_current_cdemo_sk,
. . . . . . . . . . . . > A.c_current_hdemo_sk,
. . . . . . . . . . . . > A.c_current_addr_sk,
. . . . . . . . . . . . > A.c_salutation,
. . . . . . . . . . . . > A.c_first_name,
. . . . . . . . . . . . > A.c_last_name,
. . . . . . . . . . . . > A.c_preferred_cust_flag,
. . . . . . . . . . . . > A.c_login,
. . . . . . . . . . . . > A.c_email_address,
. . . . . . . . . . . . > A.c_last_review_date,
. . . . . . . . . . . . > Sum(B.ss_quantity),
. . . . . . . . . . . . > Sum(B.ss_wholesale_cost),
. . . . . . . . . . . . > Sum(B.ss_list_price),
. . . . . . . . . . . . > Sum(B.ss_sales_price),
. . . . . . . . . . . . > Sum(B.ss_ext_discount_amt),
. . . . . . . . . . . . > Sum(B.ss_ext_sales_price),
. . . . . . . . . . . . > Sum(B.ss_ext_wholesale_cost),
. . . . . . . . . . . . > Sum(B.ss_ext_list_price),
. . . . . . . . . . . . > Sum(B.ss_ext_tax),
. . . . . . . . . . . . > Sum(B.ss_coupon_amt),
. . . . . . . . . . . . > Sum(B.ss_net_paid),
. . . . . . . . . . . . > Sum(B.ss_net_paid_inc_tax),
. . . . . . . . . . . . > Sum(B.ss_net_profit)
. . . . . . . . . . . . > FROM customer A,
. . . . . . . . . . . . >store_sales B
. . . . . . . . . . . . > WHERE
. . . . . . . . . . . . > A.c_customer_sk = B.ss_customer_sk
. . . . . . . . . . . . > GROUP BY
. . . . . . . . . . . . > A.c_customer_id,
. . . . . . . . . . . . > A.c_current_cdemo_sk,
. . . . . . . . . . . . > A.c_current_hdemo_sk,
. . . . . . . . . . . . > A.c_current_addr_sk,
. . . . . . . . . . . . > A.c_salutation,
. . . . . . . . . . . . > A.c_first_name,
. . . . . . . . . . . . > A.c_last_name,
. . . . . . . . . . . . > A.c_preferred_cust_flag,
. . . . . . . . . . . . > A.c_login,
. . . . . . . . . . . . > A.c_email_address,
. . . . . . . . . . . . > A.c_last_review_date
. . . . . . . . . . . . > LIMIT 100;
++-+-++---+---+--++--+--+-+--+-+-+-+-+-+-+-+-+-+-+-+--+
| c_customer_id | c_current_cdemo_sk | c_current_hdemo_sk |
c_current_addr_sk | c_salutation | c_first_name | c_last_name |
c_preferred_cust_flag | c_login | c_email_address | c_last_review_date |
EXPR$11 | EXPR$12 | EXPR$13 | EXPR$14 |
EXPR$15 | EXPR$16 | EXPR$17 | EXPR$18
| EXPR$19 | EXPR$20 | EXPR$21 |
EXPR$22 | EXPR$23|
++-+-++---+---+--++--+--+-+--+-+-+-+-+-+-+-+-+-+-+-+--+
| [B@1997053a| 1740699 | 6283| 8667
| [B@9d82ff6| [B@5034c486 | [B@3852628d | [B@28c433de|
null | [B@692948ba | [B@1565e146 | 1451 |
1636.3499875068665 | 2735.4400119781494 | 1026.380006607622 |
13208.909889936447 | 58845.53004384041 | 94184.37959289551 |
148164.82012939453 | 1023.8200042340904 | 13208.909889936447 |
42804.30076660216 | 46858.699785619974 | -48243.00975036621 |
| [B@53a8c5a3| 1802568 | 1029| 15490
| [B@6d477b3d | [B@6fec19d7 | [B@7594aba8 | [B@7391dee6|
null | [B@6dc10950 | [B@1c4a9b7b | 1162 |
1148.9599850177765 | 1852.3299944400787 | 777.4500068426132 |
759.4600238800049 | 35477.37022399902 | 51913.2008447 |
84324.75047302246 | 1640.790023803711 | 759.4600238800049 |
34717.91014003754 | 36358.69979476929 | -17195.299812316895 |
| [B@6625b6bf| 1744347
[jira] [Resolved] (DRILL-1611) The SQL fails Query failed: Failure while running fragment. Queue closed due to channel closure
[ https://issues.apache.org/jira/browse/DRILL-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman resolved DRILL-1611. - Resolution: Fixed Fix Version/s: (was: 1.2.0) 1.1.0 > The SQL fails Query failed: Failure while running fragment. Queue closed due > to channel closure > --- > > Key: DRILL-1611 > URL: https://issues.apache.org/jira/browse/DRILL-1611 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 0.6.0 >Reporter: Amol >Assignee: Victoria Markman > Fix For: 1.1.0 > > > Some of the sqls are executing fine. When we execute below SQL we get an > error. We doubled the memory and it does not respond. request guidance to > correct it, > SQL USED: > SELECT c_customer_id,c_current_cdemo_sk,c_current_hdemo_sk,c_current_addr_sk, > c_salutation,c_first_name,c_last_name,c_preferred_cust_flag, > c_login,c_email_address,c_last_review_date,Sum(ss_quantity), > Sum(ss_wholesale_cost),Sum(ss_list_price),Sum(ss_sales_price), > Sum(ss_ext_discount_amt),Sum(ss_ext_sales_price),Sum(ss_ext_wholesale_cost), > Sum(ss_ext_list_price),Sum(ss_ext_tax),Sum(ss_coupon_amt),Sum(ss_net_paid), > Sum(ss_net_paid_inc_tax),Sum(ss_net_profit) > from customer A , store_sales B where A.c_customer_sk = B.ss_customer_sk > Group by > c_customer_id,c_current_cdemo_sk,c_current_hdemo_sk,c_current_addr_sk, > c_salutation,c_first_name,c_last_name,c_preferred_cust_flag, > c_login,c_email_address,c_last_review_date > limit 100; > ERROR: > Query failed: Failure while running fragment. Queue closed due to channel > closure. [9a718ee0-2f6b-401f-b4d6-b861ef4769da] > Error: exception while executing query: Failure while trying to get next > result batch. (state=,code=0) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2247) SUM with CASE statement on the column of the null producing side of left outer join returns wrong result
[
https://issues.apache.org/jira/browse/DRILL-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615727#comment-14615727
]
Victoria Markman commented on DRILL-2247:
-
1. column c1 in x1.parquet and x2.parquet is of decimal data type (before it
was disabled in 1.0)
2. I can't reproduce this anymore: tried to force the plan that is posted in
the bug by disabling merge join and setting `planner.slice_target` = 1 and I
get correct result.
With 1.1 I get:
Default plan with all the default settings (result is correct)
{code}
0: jdbc:drill:schema=dfs> explain plan for select
. . . . . . . . . . . . > count(*) as count_star,
. . . . . . . . . . . . > sum(x2.c1) as simle_sum,
. . . . . . . . . . . . > round(avg(x2.c1)) as round_avg_x2_c1,
. . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
round(x2.c1/12) end) as sum_with_case
. . . . . . . . . . . . > from
. . . . . . . . . . . . > x1 left outer join x2 on x1.c1 = x2.c1
. . . . . . . . . . . . > group by
. . . . . . . . . . . . > x1.c1;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(count_star=[$0], simle_sum=[$1], round_avg_x2_c1=[$2],
sum_with_case=[$3])
00-02Project(count_star=[$1], simle_sum=[CASE(=($3, 0), null, $2)],
round_avg_x2_c1=[ROUND(CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY NOT
NULL)], sum_with_case=[CASE(=($5, 0), null, $4)])
00-03 HashAgg(group=[{0}], count_star=[COUNT()], agg#1=[$SUM0($1)],
agg#2=[COUNT($1)], agg#3=[$SUM0($2)], agg#4=[COUNT($2)])
00-04Project(c1=[$0], c10=[$1], $f2=[CASE(=($1, 0), CAST(100):ANY,
ROUND(/($1, 12)))])
00-05 Project(c1=[$1], c10=[$0])
00-06MergeJoin(condition=[=($1, $0)], joinType=[right])
00-08 SelectionVectorRemover
00-10Sort(sort0=[$0], dir0=[ASC])
00-12 Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/x2]],
selectionRoot=maprfs:/drill/testdata/subqueries/x2, numFiles=1,
columns=[`c1`]]])
00-07 Project(c10=[$0])
00-09SelectionVectorRemover
00-11 Sort(sort0=[$0], dir0=[ASC])
00-13Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath [path=maprfs:///drill/testdata/subqueries/x1]],
selectionRoot=maprfs:/drill/testdata/subqueries/x1, numFiles=1,
columns=[`c1`]]])
{code}
With merge join disabled (correct result):
{code}
0: jdbc:drill:schema=dfs> select
. . . . . . . . . . . . > count(*) as count_star,
. . . . . . . . . . . . > sum(x2.c1) as simle_sum,
. . . . . . . . . . . . > round(avg(x2.c1)) as round_avg_x2_c1,
. . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
round(x2.c1/12) end) as sum_with_case
. . . . . . . . . . . . > from
. . . . . . . . . . . . > `x1.parquet` x1 left outer join `x2.parquet`
x2 on x1.c1 = x2.c1
. . . . . . . . . . . . > group by
. . . . . . . . . . . . > x1.c1;
+-++--++
| count_star | simle_sum | round_avg_x2_c1 | sum_with_case |
+-++--++
| 1 | null | null | null |
+-++--++
1 row selected (0.517 seconds)
0: jdbc:drill:schema=dfs> explain plan for select
. . . . . . . . . . . . > count(*) as count_star,
. . . . . . . . . . . . > sum(x2.c1) as simle_sum,
. . . . . . . . . . . . > round(avg(x2.c1)) as round_avg_x2_c1,
. . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
round(x2.c1/12) end) as sum_with_case
. . . . . . . . . . . . > from
. . . . . . . . . . . . > `x1.parquet` x1 left outer join `x2.parquet`
x2 on x1.c1 = x2.c1
. . . . . . . . . . . . > group by
. . . . . . . . . . . . > x1.c1;
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(count_star=[$0], simle_sum=[$1], round_avg_x2_c1=[$2],
sum_with_case=[$3])
00-02Project(count_star=[$1], simle_sum=[CASE(=($3, 0), null, $2)],
round_avg_x2_c1=[ROUND(CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY NOT
NULL)], sum_with_case=[CASE(=($5, 0), null, $4)])
00-03 HashAgg(group=[{0}], count_star=[COUNT()], agg#1=[$SUM0($1)],
agg#2=[COUNT($1)], agg#3=[$SUM0($2)], agg#4=[COUNT($2)])
00-04Project(c1=[$0], c10=[$1], $f2=[CASE(=($1, 0), CAST(100):ANY,
ROUND(/($1, 12)))])
00-05 Project(c1=[$1
[jira] [Commented] (DRILL-2045) NPE when querying INFORMATION_SCHEMA table
[ https://issues.apache.org/jira/browse/DRILL-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615731#comment-14615731 ] Victoria Markman commented on DRILL-2045: - I did not see this issue since it was reported, closing it as "Can't reproduce." > NPE when querying INFORMATION_SCHEMA table > -- > > Key: DRILL-2045 > URL: https://issues.apache.org/jira/browse/DRILL-2045 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Information Schema >Affects Versions: 0.8.0 >Reporter: Victoria Markman >Assignee: Victoria Markman > Attachments: drill-2045.log > > > I'm filing this bug as a placeholder to investigate the reason of NPE on > "select * from INFORMATION_SCHEMA.`TABLES`" on my cluster. > My strong suspicion is that it is related to the way I upgrade drill on a > cluster and this is somehow scrambles zookeeper's state. > My steps are: > * stop drillbits > * uninstall drill rpm: "rpm -e <>" > * install a new rpm: "rpm -i <>" > * service mapr-warden restart (I don't stop warden before an upgrade) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-2045) NPE when querying INFORMATION_SCHEMA table
[ https://issues.apache.org/jira/browse/DRILL-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman closed DRILL-2045. --- > NPE when querying INFORMATION_SCHEMA table > -- > > Key: DRILL-2045 > URL: https://issues.apache.org/jira/browse/DRILL-2045 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Information Schema >Affects Versions: 0.8.0 >Reporter: Victoria Markman >Assignee: Victoria Markman > Attachments: drill-2045.log > > > I'm filing this bug as a placeholder to investigate the reason of NPE on > "select * from INFORMATION_SCHEMA.`TABLES`" on my cluster. > My strong suspicion is that it is related to the way I upgrade drill on a > cluster and this is somehow scrambles zookeeper's state. > My steps are: > * stop drillbits > * uninstall drill rpm: "rpm -e <>" > * install a new rpm: "rpm -i <>" > * service mapr-warden restart (I don't stop warden before an upgrade) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2045) NPE when querying INFORMATION_SCHEMA table
[ https://issues.apache.org/jira/browse/DRILL-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman resolved DRILL-2045. - Resolution: Cannot Reproduce Fix Version/s: (was: 1.2.0) > NPE when querying INFORMATION_SCHEMA table > -- > > Key: DRILL-2045 > URL: https://issues.apache.org/jira/browse/DRILL-2045 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Information Schema >Affects Versions: 0.8.0 >Reporter: Victoria Markman >Assignee: Victoria Markman > Attachments: drill-2045.log > > > I'm filing this bug as a placeholder to investigate the reason of NPE on > "select * from INFORMATION_SCHEMA.`TABLES`" on my cluster. > My strong suspicion is that it is related to the way I upgrade drill on a > cluster and this is somehow scrambles zookeeper's state. > My steps are: > * stop drillbits > * uninstall drill rpm: "rpm -e <>" > * install a new rpm: "rpm -i <>" > * service mapr-warden restart (I don't stop warden before an upgrade) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2247) SUM with CASE statement on the column of the null producing side of left outer join returns wrong result
[
https://issues.apache.org/jira/browse/DRILL-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman resolved DRILL-2247.
-
Resolution: Fixed
Fix Version/s: (was: 1.2.0)
1.1.0
> SUM with CASE statement on the column of the null producing side of left
> outer join returns wrong result
>
>
> Key: DRILL-2247
> URL: https://issues.apache.org/jira/browse/DRILL-2247
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.8.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
> Fix For: 1.1.0
>
> Attachments: x1.parquet, x2.parquet
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select * from x1;
> ++
> | c1 |
> ++
> | 1032.6516 |
> ++
> 1 row selected (0.093 seconds)
> 0: jdbc:drill:schema=dfs> select * from x2;
> ++
> | c1 |
> ++
> | 1057.3117 |
> | 1090.8299 |
> ++
> 2 rows selected (0.085 seconds)
> {code}
> Correct result is NULL:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > sum(x2.c1)
>as simple_sum,
> . . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
> round(x2.c1/12) end) as sum_with_case
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > x1 left outer join x2 on x1.c1 = x2.c1
> . . . . . . . . . . . . > group by
> . . . . . . . . . . . . > x1.c1;
> ++---+
> | simple_sum | sum_with_case |
> ++---+
> | 0. | null |
> ++---+
> 1 row selected (0.173 seconds)
> {code}
> Query plan for correct result:
> {code}
> 00-01 Project(simple_sum=[$0], sum_with_case=[$1])
> 00-02UnionExchange
> 01-01 Project(simple_sum=[$1], sum_with_case=[$2])
> 01-02HashAgg(group=[{0}], simple_sum=[SUM($1)],
> sum_with_case=[SUM($2)])
> 01-03 Project(c1=[$0], c10=[$1], $f2=[CASE(=($1, 0),
> CAST(100):ANY NOT NULL, ROUND(/($1, 12)))])
> 01-04HashJoin(condition=[=($0, $1)], joinType=[left])
> 01-06 HashToRandomExchange(dist0=[[$0]])
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/aggregation/x1]],
> selectionRoot=/aggregation/x1, numFiles=1, columns=[`c1`]]])
> 01-05 Project(c10=[$0])
> 01-07HashToRandomExchange(dist0=[[$0]])
> 03-01 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/aggregation/x2]],
> selectionRoot=/aggregation/x2, numFiles=1, columns=[`c1`]]])
> {code}
> Wrong result: sum with case returns 0.0 instead of NULL
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*)
>as count_star,
> . . . . . . . . . . . . > sum(x2.c1)
>as simle_sum,
> . . . . . . . . . . . . > round(avg(x2.c1))
>as round_avg_x2_c1,
> . . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
> round(x2.c1/12) end) as sum_with_case
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > x1 left outer join x2 on x1.c1 = x2.c1
> . . . . . . . . . . . . > group by
> . . . . . . . . . . . . > x1.c1;
> +++-+---+
> | count_star | simle_sum | round_avg_x2_c1 | sum_with_case |
> +++-+---+
> | 1 | null | null| 0.0 |
> +++-+---+
> {code}
> Query plan for the wrong result query:
> {code}
> 00-01 Project(count_star=[$0], simle_sum=[$1], round_avg_x2_c1=[$2],
> sum_with_case=[$3])
> 00-02UnionExchange
> 01-01 Project(count_star=[$1], simle_sum=[CASE(=($3, 0), null, $2)],
> round_avg_x2_c1=[ROUND(CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY)],
> sum_with_case=[$4])
> 01-02HashAgg(group=[{0}], count_star=[COUNT()],
> agg#1=[$SUM0($1)], agg#2=[COUNT($1)], agg#3=[$SUM0($2)])
> 01-03 Project(c1=[$0], c10=[$1], $f2=[CASE(=($1, 0),
> CAST(100):ANY NOT NULL, ROUND(/($1, 12)))])
> 01-04HashJoin(condition=[=($0, $1)], joinType=[left])
> 01-06 HashToRandomExchange(dist0=[[$0]])
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/aggregation/x1]],
> selectionRoot=/aggregation/x1, numFiles=1, columns=[`c1`]]])
> 01-05
[jira] [Closed] (DRILL-2247) SUM with CASE statement on the column of the null producing side of left outer join returns wrong result
[
https://issues.apache.org/jira/browse/DRILL-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2247.
---
> SUM with CASE statement on the column of the null producing side of left
> outer join returns wrong result
>
>
> Key: DRILL-2247
> URL: https://issues.apache.org/jira/browse/DRILL-2247
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.8.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
> Fix For: 1.1.0
>
> Attachments: x1.parquet, x2.parquet
>
>
> {code}
> 0: jdbc:drill:schema=dfs> select * from x1;
> ++
> | c1 |
> ++
> | 1032.6516 |
> ++
> 1 row selected (0.093 seconds)
> 0: jdbc:drill:schema=dfs> select * from x2;
> ++
> | c1 |
> ++
> | 1057.3117 |
> | 1090.8299 |
> ++
> 2 rows selected (0.085 seconds)
> {code}
> Correct result is NULL:
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > sum(x2.c1)
>as simple_sum,
> . . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
> round(x2.c1/12) end) as sum_with_case
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > x1 left outer join x2 on x1.c1 = x2.c1
> . . . . . . . . . . . . > group by
> . . . . . . . . . . . . > x1.c1;
> ++---+
> | simple_sum | sum_with_case |
> ++---+
> | 0. | null |
> ++---+
> 1 row selected (0.173 seconds)
> {code}
> Query plan for correct result:
> {code}
> 00-01 Project(simple_sum=[$0], sum_with_case=[$1])
> 00-02UnionExchange
> 01-01 Project(simple_sum=[$1], sum_with_case=[$2])
> 01-02HashAgg(group=[{0}], simple_sum=[SUM($1)],
> sum_with_case=[SUM($2)])
> 01-03 Project(c1=[$0], c10=[$1], $f2=[CASE(=($1, 0),
> CAST(100):ANY NOT NULL, ROUND(/($1, 12)))])
> 01-04HashJoin(condition=[=($0, $1)], joinType=[left])
> 01-06 HashToRandomExchange(dist0=[[$0]])
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/aggregation/x1]],
> selectionRoot=/aggregation/x1, numFiles=1, columns=[`c1`]]])
> 01-05 Project(c10=[$0])
> 01-07HashToRandomExchange(dist0=[[$0]])
> 03-01 Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/aggregation/x2]],
> selectionRoot=/aggregation/x2, numFiles=1, columns=[`c1`]]])
> {code}
> Wrong result: sum with case returns 0.0 instead of NULL
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > count(*)
>as count_star,
> . . . . . . . . . . . . > sum(x2.c1)
>as simle_sum,
> . . . . . . . . . . . . > round(avg(x2.c1))
>as round_avg_x2_c1,
> . . . . . . . . . . . . > sum(case when x2.c1 = 0 then 100 else
> round(x2.c1/12) end) as sum_with_case
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > x1 left outer join x2 on x1.c1 = x2.c1
> . . . . . . . . . . . . > group by
> . . . . . . . . . . . . > x1.c1;
> +++-+---+
> | count_star | simle_sum | round_avg_x2_c1 | sum_with_case |
> +++-+---+
> | 1 | null | null| 0.0 |
> +++-+---+
> {code}
> Query plan for the wrong result query:
> {code}
> 00-01 Project(count_star=[$0], simle_sum=[$1], round_avg_x2_c1=[$2],
> sum_with_case=[$3])
> 00-02UnionExchange
> 01-01 Project(count_star=[$1], simle_sum=[CASE(=($3, 0), null, $2)],
> round_avg_x2_c1=[ROUND(CAST(/(CastHigh(CASE(=($3, 0), null, $2)), $3)):ANY)],
> sum_with_case=[$4])
> 01-02HashAgg(group=[{0}], count_star=[COUNT()],
> agg#1=[$SUM0($1)], agg#2=[COUNT($1)], agg#3=[$SUM0($2)])
> 01-03 Project(c1=[$0], c10=[$1], $f2=[CASE(=($1, 0),
> CAST(100):ANY NOT NULL, ROUND(/($1, 12)))])
> 01-04HashJoin(condition=[=($0, $1)], joinType=[left])
> 01-06 HashToRandomExchange(dist0=[[$0]])
> 02-01Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:/aggregation/x1]],
> selectionRoot=/aggregation/x1, numFiles=1, columns=[`c1`]]])
> 01-05 Project(c10=[$0])
> 01-07HashToRandomExchange(dist0=
[jira] [Commented] (DRILL-2710) NPE in a regression run of joins/order_by/queries/q16.sql on an 8 node cluster
[
https://issues.apache.org/jira/browse/DRILL-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615748#comment-14615748
]
Victoria Markman commented on DRILL-2710:
-
This test has been running our framework since reported exception successfully.
I made couple of attempts to reproduce this unsuccessfully. Marking it as
resolved. Please reopen if you run into this again.
> NPE in a regression run of joins/order_by/queries/q16.sql on an 8 node cluster
> --
>
> Key: DRILL-2710
> URL: https://issues.apache.org/jira/browse/DRILL-2710
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
> Fix For: 1.1.0
>
>
> Assigning to myself to debug. Will change summary as soon as I figure out the
> easier way to reproduce the problem.
> {code}
> /root/drillAutomation/framework/framework/resources/Precommit/Functional/joins/order_by/queries/q16.sql
> Query:
> select count(*),
> count(sq1.a1),
> count(sq2.a2),
> min(sq1.a1),
> max(sq1.a1),
> min(sq2.a2),
> max(sq2.a2),
> avg(sq1.a1),
> avg(sq2.a2)
> from
> (select c_integer, c_date from j1 order by c_integer asc nulls first)
> as sq1(a1, b1)
> inner join
> (select c_integer, c_date from j2 order by c_integer asc) as sq2(a2,
> b2)
> on (sq1.a1 = sq2.a2)
> Failed with exception
> java.sql.SQLException: exception while executing query: Failure while
> executing query.
> at net.hydromatic.avatica.Helper.createException(Helper.java:40)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
> at
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:139)
> at
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:80)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.sql.SQLException: Failure while executing query.
> at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> ... 9 more
> Caused by: org.apache.drill.exec.rpc.RpcException: NullPointerException:
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
> at
> org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(Chan
[jira] [Closed] (DRILL-2710) NPE in a regression run of joins/order_by/queries/q16.sql on an 8 node cluster
[
https://issues.apache.org/jira/browse/DRILL-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2710.
---
> NPE in a regression run of joins/order_by/queries/q16.sql on an 8 node cluster
> --
>
> Key: DRILL-2710
> URL: https://issues.apache.org/jira/browse/DRILL-2710
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
> Fix For: 1.1.0
>
>
> Assigning to myself to debug. Will change summary as soon as I figure out the
> easier way to reproduce the problem.
> {code}
> /root/drillAutomation/framework/framework/resources/Precommit/Functional/joins/order_by/queries/q16.sql
> Query:
> select count(*),
> count(sq1.a1),
> count(sq2.a2),
> min(sq1.a1),
> max(sq1.a1),
> min(sq2.a2),
> max(sq2.a2),
> avg(sq1.a1),
> avg(sq2.a2)
> from
> (select c_integer, c_date from j1 order by c_integer asc nulls first)
> as sq1(a1, b1)
> inner join
> (select c_integer, c_date from j2 order by c_integer asc) as sq2(a2,
> b2)
> on (sq1.a1 = sq2.a2)
> Failed with exception
> java.sql.SQLException: exception while executing query: Failure while
> executing query.
> at net.hydromatic.avatica.Helper.createException(Helper.java:40)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
> at
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:139)
> at
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:80)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.sql.SQLException: Failure while executing query.
> at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> ... 9 more
> Caused by: org.apache.drill.exec.rpc.RpcException: NullPointerException:
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
> at
> org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319
[jira] [Resolved] (DRILL-2710) NPE in a regression run of joins/order_by/queries/q16.sql on an 8 node cluster
[
https://issues.apache.org/jira/browse/DRILL-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman resolved DRILL-2710.
-
Resolution: Fixed
Fix Version/s: (was: 1.2.0)
1.1.0
> NPE in a regression run of joins/order_by/queries/q16.sql on an 8 node cluster
> --
>
> Key: DRILL-2710
> URL: https://issues.apache.org/jira/browse/DRILL-2710
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Victoria Markman
> Fix For: 1.1.0
>
>
> Assigning to myself to debug. Will change summary as soon as I figure out the
> easier way to reproduce the problem.
> {code}
> /root/drillAutomation/framework/framework/resources/Precommit/Functional/joins/order_by/queries/q16.sql
> Query:
> select count(*),
> count(sq1.a1),
> count(sq2.a2),
> min(sq1.a1),
> max(sq1.a1),
> min(sq2.a2),
> max(sq2.a2),
> avg(sq1.a1),
> avg(sq2.a2)
> from
> (select c_integer, c_date from j1 order by c_integer asc nulls first)
> as sq1(a1, b1)
> inner join
> (select c_integer, c_date from j2 order by c_integer asc) as sq2(a2,
> b2)
> on (sq1.a1 = sq2.a2)
> Failed with exception
> java.sql.SQLException: exception while executing query: Failure while
> executing query.
> at net.hydromatic.avatica.Helper.createException(Helper.java:40)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:406)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:78)
> at
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:139)
> at
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:80)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.sql.SQLException: Failure while executing query.
> at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:105)
> at org.apache.drill.jdbc.DrillResultSet.execute(DrillResultSet.java:44)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> ... 9 more
> Caused by: org.apache.drill.exec.rpc.RpcException: NullPointerException:
> at
> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111)
> at
> org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52)
> at
> org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34)
> at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57)
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
> at
> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> at
> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> at
> io.netty.ch
[jira] [Created] (DRILL-3470) Add support for multiple partition by clauses for window functions in the same query
Victoria Markman created DRILL-3470: --- Summary: Add support for multiple partition by clauses for window functions in the same query Key: DRILL-3470 URL: https://issues.apache.org/jira/browse/DRILL-3470 Project: Apache Drill Issue Type: New Feature Components: Query Planning & Optimization Reporter: Victoria Markman Assignee: Jinfeng Ni We disabled support for multiple partitions in 1.1. (DRILL-3196) This is a request to enable and productize multiple partitions for window functions in the same sql statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3470) Add support for multiple partition by clauses for window functions in the same query
[ https://issues.apache.org/jira/browse/DRILL-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-3470: Labels: window_funcion (was: ) > Add support for multiple partition by clauses for window functions in the > same query > > > Key: DRILL-3470 > URL: https://issues.apache.org/jira/browse/DRILL-3470 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Jinfeng Ni > Labels: window_funcion > > We disabled support for multiple partitions in 1.1. (DRILL-3196) > This is a request to enable and productize multiple partitions for window > functions in the same sql statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3470) Add support for multiple partition by clauses for window functions in the same query
[ https://issues.apache.org/jira/browse/DRILL-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-3470: Labels: window_function (was: window_funcion) > Add support for multiple partition by clauses for window functions in the > same query > > > Key: DRILL-3470 > URL: https://issues.apache.org/jira/browse/DRILL-3470 > Project: Apache Drill > Issue Type: New Feature > Components: Query Planning & Optimization >Reporter: Victoria Markman >Assignee: Jinfeng Ni > Labels: window_function > > We disabled support for multiple partitions in 1.1. (DRILL-3196) > This is a request to enable and productize multiple partitions for window > functions in the same sql statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2967:
Fix Version/s: (was: 1.1.0)
1.2.0
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2 where b2 not in (select b1 from
> t1);
> ++++
> | a1 | b1 | c1 |
> ++++
> Query failed: SYSTEM ERROR: Join only supports implicit casts between 1.
> Numeric data
> 2. Varchar, Varbinary data Left type: INT, Right type: VARCHAR. Add explicit
> casts to avoid this error
> Fragment 1:0
> [1a83aa50-39aa-452c-91dd-970bf4a8f03d on atsqa4-133.qa.lab:31010]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing
> query.
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> at sqlline.SqlLine.print(SqlLine.java:1809)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
> at sqlline.SqlLine.dispatch(SqlLine.java:889)
> at sqlline.SqlLine.begin(SqlLine.java:763)
> at sqlline.SqlLine.start(SqlLine.java:498)
> at sqlline.SqlLine.main(SqlLine.java:460)
> {code}
> Explain plan for the query with an error:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select * from t2 where b2 not in
> (select b1 from t1);
> +++
> | text | json |
> +++
> | 00-00 Screen
> 00-01 Project(*=[$0])
> 00-02 UnionExchange
> 01-01 Project(T27¦¦*=[$0])
> 01-02 SelectionVectorRemover
> 01-03 Filter(condition=[NOT(CASE(=($2, 0), false, IS NOT NULL($6), true, IS
> NULL($4), null, <($3, $2), null, false))])
> 01-04 MergeJoin(condition=[=($4, $5)], joinType=[left])
> 01-06 SelectionVectorRemover
> 01-08 Sort(sort0=[$4], dir0=[ASC])
> 01-10 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4])
> 01-12 HashToRandomExchange(dist0=[[$4]])
> 02-01 UnorderedMuxExchange
> 04-01 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4],
> E_X_P_R_H_A_S_H_F_I_E_L_D=[castInt(hash64AsDouble($4))])
[jira] [Reopened] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman reopened DRILL-2967:
-
I can still reproduce this issue with:
#Thu Jul 09 02:11:10 EDT 2015
git.commit.id.abbrev=72f9469
{code}
0: jdbc:drill:schema=dfs> select * from j2 where c_bigint not in ( select
cast(c_integer as bigint) from j6 where c_boolean is true ) and c_date not in (
select distinct c_date from j6);
java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
DrillRuntimeException: Join only supports implicit casts between 1. Numeric data
2. Varchar, Varbinary data 3. Date, Timestamp data Left type: DATE, Right
type: INT. Add explicit casts to avoid this error
Fragment 1:2
[Error Id: bada137a-957a-492c-af77-311b5830bf01 on atsqa4-133.qa.lab:31010]
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
at sqlline.SqlLine.print(SqlLine.java:1583)
at sqlline.Commands.execute(Commands.java:852)
at sqlline.Commands.sql(Commands.java:751)
at sqlline.SqlLine.dispatch(SqlLine.java:738)
at sqlline.SqlLine.begin(SqlLine.java:612)
at sqlline.SqlLine.start(SqlLine.java:366)
at sqlline.SqlLine.main(SqlLine.java:259)
{code}
Will attach reproduction shortly.
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.2.0
>
> Attachments: t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2 where b2 not in (select b1 from
> t1);
> ++++
> | a1 | b1 | c1 |
> ++++
> Query failed: SYSTEM ERROR: Join only supports implicit casts between 1.
> Numeric data
> 2. Varchar, Varbinary data Left type: INT, Right type: VARCHAR. Add explicit
> casts to avoid this error
> Fragment 1:0
> [1a83aa50-39aa-452c-91dd-970bf4a8f03d on atsqa4-133.qa.lab:31010]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing
> query.
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> at sqlline.SqlLine.print(SqlLine.java:1809)
>
[jira] [Commented] (DRILL-3898) No space error during external sort does not cancel the query
[
https://issues.apache.org/jira/browse/DRILL-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461493#comment-15461493
]
Victoria Markman commented on DRILL-3898:
-
[~ben-zvi] nice.
> No space error during external sort does not cancel the query
> -
>
> Key: DRILL-3898
> URL: https://issues.apache.org/jira/browse/DRILL-3898
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.2.0, 1.8.0
>Reporter: Victoria Markman
>Assignee: Boaz Ben-Zvi
> Fix For: Future
>
> Attachments: drillbit.log, sqlline_3898.ver_1_8.log
>
>
> While verifying DRILL-3732 I ran into a new problem.
> I think drill somehow loses track of out of disk exception and does not
> cancel rest of the query, which results in NPE:
> Reproduction is the same as in DRILL-3732:
> {code}
> 0: jdbc:drill:schema=dfs> create table store_sales_20(ss_item_sk,
> ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk)
> partition by (ss_promo_sk) as
> . . . . . . . . . . . . > select
> . . . . . . . . . . . . > case when columns[2] = '' then cast(null as
> varchar(100)) else cast(columns[2] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[3] = '' then cast(null as
> varchar(100)) else cast(columns[3] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[4] = '' then cast(null as
> varchar(100)) else cast(columns[4] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[5] = '' then cast(null as
> varchar(100)) else cast(columns[5] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[0] = '' then cast(null as
> varchar(100)) else cast(columns[0] as varchar(100)) end,
> . . . . . . . . . . . . > case when columns[8] = '' then cast(null as
> varchar(100)) else cast(columns[8] as varchar(100)) end
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > `store_sales.dat` ss
> . . . . . . . . . . . . > ;
> Error: SYSTEM ERROR: NullPointerException
> Fragment 1:16
> [Error Id: 0ae9338d-d04f-4b4a-93aa-a80d13cedb29 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> This exception in drillbit.log should have triggered query cancellation:
> {code}
> 2015-10-06 17:01:34,463 [WorkManager-2] ERROR
> o.apache.drill.exec.work.WorkManager -
> org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an exception.
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:226)
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.7.0_71]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.7.0_71]
> at java.io.FilterOutputStream.close(FilterOutputStream.java:157)
> ~[na:1.7.0_71]
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:400)
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
> ~[hadoop-common-2.5.1-mapr-1503.jar:na]
> at
> org.apache.drill.exec.physical.impl.xsort.BatchGroup.close(BatchGroup.java:152)
> ~[drill-java-exec-1.2.0.jar:1.2.0]
> at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:44)
> ~[drill-common-1.2.0.jar:1.2.0]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:553)
> ~[drill-java-exec-1.2.0.jar:1.2.0]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:362)
> ~[drill-java-exec-1.2.0.jar:1.2.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147)
> ~[drill-java-exec-1.2.0.jar:1.2.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:104)
> ~[drill-java-exec-1.2.0.jar:1.2.0]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:94)
> ~[drill-java-exec-1.2.0.jar:1.2.0]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:
[jira] [Updated] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-2967:
Reviewer: Victoria Markman
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
> Fix For: 1.3.0
>
> Attachments: j2_j6_tables.tar, t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2 where b2 not in (select b1 from
> t1);
> ++++
> | a1 | b1 | c1 |
> ++++
> Query failed: SYSTEM ERROR: Join only supports implicit casts between 1.
> Numeric data
> 2. Varchar, Varbinary data Left type: INT, Right type: VARCHAR. Add explicit
> casts to avoid this error
> Fragment 1:0
> [1a83aa50-39aa-452c-91dd-970bf4a8f03d on atsqa4-133.qa.lab:31010]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing
> query.
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> at sqlline.SqlLine.print(SqlLine.java:1809)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
> at sqlline.SqlLine.dispatch(SqlLine.java:889)
> at sqlline.SqlLine.begin(SqlLine.java:763)
> at sqlline.SqlLine.start(SqlLine.java:498)
> at sqlline.SqlLine.main(SqlLine.java:460)
> {code}
> Explain plan for the query with an error:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select * from t2 where b2 not in
> (select b1 from t1);
> +++
> | text | json |
> +++
> | 00-00 Screen
> 00-01 Project(*=[$0])
> 00-02 UnionExchange
> 01-01 Project(T27¦¦*=[$0])
> 01-02 SelectionVectorRemover
> 01-03 Filter(condition=[NOT(CASE(=($2, 0), false, IS NOT NULL($6), true, IS
> NULL($4), null, <($3, $2), null, false))])
> 01-04 MergeJoin(condition=[=($4, $5)], joinType=[left])
> 01-06 SelectionVectorRemover
> 01-08 Sort(sort0=[$4], dir0=[ASC])
> 01-10 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4])
> 01-12 HashToRandomExchange(dist0=[[$4]])
> 02-01 UnorderedMuxExchange
> 04-01 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4],
> E_X_P_R_H_A_S_H_F_I_E_L_D=[castInt(hash64AsDouble($4))])
> 04-02 Proj
[jira] [Updated] (DRILL-4125) Illegal argument exception during merge join
[
https://issues.apache.org/jira/browse/DRILL-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4125:
Reviewer: Victoria Markman
> Illegal argument exception during merge join
> -
>
> Key: DRILL-4125
> URL: https://issues.apache.org/jira/browse/DRILL-4125
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac59f2-5d92-7378-bf81-e844a300efd7.sys.drill,
> drillbit.log
>
>
> Same setup as in DRILL-4109
> Query: framework/resources/Advanced/tpcds/tpcds_sf100/original/query93.sql
> Excerpt from drillbit.log
> {code}
> 2015-11-23 23:50:44,071 [29ac59f2-5d92-7378-bf81-e844a300efd7:frag:5:74]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalArgumentException
> Fragment 5:74
> [Error Id: 1ca9758d-1864-4940-9efa-b8906d4f9b52 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException
> Fragment 5:74
> [Error Id: 1ca9758d-1864-4940-9efa-b8906d4f9b52 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: null
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> ~[guava-14.0.1.jar:na]
> at
> org.apache.drill.exec.record.RecordIterator.getCurrentPosition(RecordIterator.java:242)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.JoinWorkerGen8348.doJoin(JoinTemplate.java:63)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:206)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:276)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-
[jira] [Updated] (DRILL-4109) NPE in RecordIterator
[
https://issues.apache.org/jira/browse/DRILL-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4109:
Reviewer: Victoria Markman
> NPE in RecordIterator
> -
>
> Key: DRILL-4109
> URL: https://issues.apache.org/jira/browse/DRILL-4109
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac6c1b-9b33-3457-8bc8-9e2dff6ad438.sys.drill,
> 29b41f37-4803-d7ce-e05f-912d1f65da79.sys.drill, drillbit.log,
> drillbit.log.debug
>
>
> 4 node cluster
> 36GB of direct memory
> 4GB heap memory
> planner.memory.max_query_memory_per_node=2GB (default)
> planner.enable_hashjoin = false
> Spill directory has 6.4T of memory available:
> {noformat}
> [Tue Nov 17 18:23:18 /tmp/drill ] # df -H .
> Filesystem Size Used Avail Use% Mounted on
> localhost:/mapr 7.7T 1.4T 6.4T 18% /mapr
> {noformat}
> Run query below:
> framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> drillbit.log
> {code}
> 2015-11-18 02:22:12,639 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:9] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_9/operator_17/7
> 2015-11-18 02:22:12,770 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:5] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_5/operator_17/7
> 2015-11-18 02:22:13,345 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:17] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_17/operator_17/7
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_13/operator_16/1
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 34 batch groups.
> Current allocated memory: 2252186
> 2015-11-18 02:22:13,363 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested RUNNING -->
> FAILED
> 2015-11-18 02:22:13,370 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested FAILED -->
> FINISHED
> 2015-11-18 02:22:13,371 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> java.lang.NullPointerException: null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3802) Throw unsupported error for ROLLUP/GROUPING
[
https://issues.apache.org/jira/browse/DRILL-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3802:
Reviewer: Victoria Markman
> Throw unsupported error for ROLLUP/GROUPING
> ---
>
> Key: DRILL-3802
> URL: https://issues.apache.org/jira/browse/DRILL-3802
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.3.0
>
>
> I believe that this is the cause of assertions in TPCDS #36, #67
> {code}
> SELECT Sum(ss_net_profit) / Sum(ss_ext_sales_price) AS
> gross_margin,
> i_category,
> i_class,
> Grouping(i_category) + Grouping(i_class) AS
> lochierarchy,
> Rank()
> OVER (
> partition BY Grouping(i_category)+Grouping(i_class), CASE
> WHEN Grouping(
> i_class) = 0 THEN i_category END
> ORDER BY Sum(ss_net_profit)/Sum(ss_ext_sales_price) ASC) AS
> rank_within_parent
> FROM store_sales,
> date_dim d1,
> item,
> store
> WHERE d1.d_year = 2000
> AND d1.d_date_sk = ss_sold_date_sk
> AND i_item_sk = ss_item_sk
> AND s_store_sk = ss_store_sk
> AND s_state IN ( 'TN', 'TN', 'TN', 'TN',
> 'TN', 'TN', 'TN', 'TN' )
> GROUP BY rollup( i_category, i_class )
> ORDER BY lochierarchy DESC,
> CASE
> WHEN lochierarchy = 0 THEN i_category
> END,
> rank_within_parent
> LIMIT 100;
> Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated:
> conversion result not null
> [Error Id: 6afae7ce-c426-44f3-a600-aa34ab7632a1 on ucs-node5.perf.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: AssertionError: Internal error:
> invariant violated: conversion result not null
> [Error Id: 6afae7ce-c426-44f3-a600-aa34ab7632a1 on ucs-node5.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1359)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:74)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at
> net.hydromatic.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:338)
> at net.hydromatic.avatica.AvaticaStatement.execute(AvaticaStatement.java:69)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:86)
> at sqlline.Commands.execute(Commands.java:841)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.runCommands(SqlLine.java:1641)
> at sqlline.Commands.run(Commands.java:1304)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:734)
> at sqlline.SqlLine.initArgs(SqlLine.java:544)
> at sqlline.SqlLine.begin(SqlLine.java:587)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: AssertionError: Internal error: invariant violated: conversion result
> not null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3871) Off by one error while reading binary fields with one terminal null in parquet
[
https://issues.apache.org/jira/browse/DRILL-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3871:
Reviewer: Victoria Markman
> Off by one error while reading binary fields with one terminal null in parquet
> --
>
> Key: DRILL-3871
> URL: https://issues.apache.org/jira/browse/DRILL-3871
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.2.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: int96
> Fix For: 1.3.0
>
> Attachments: tables.tar
>
>
> Both tables in the join where created by impala, with column c_timestamp
> being parquet int96.
> {code}
> 0: jdbc:drill:schema=dfs> select
> . . . . . . . . . . . . > max(t1.c_timestamp),
> . . . . . . . . . . . . > min(t1.c_timestamp),
> . . . . . . . . . . . . > count(t1.c_timestamp)
> . . . . . . . . . . . . > from
> . . . . . . . . . . . . > imp_t1 t1
> . . . . . . . . . . . . > inner join
> . . . . . . . . . . . . > imp_t2 t2
> . . . . . . . . . . . . > on (t1.c_timestamp = t2.c_timestamp)
> . . . . . . . . . . . . > ;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR:
> TProtocolException: Required field 'uncompressed_page_size' was not found in
> serialized data! Struct: PageHeader(type:null, uncompressed_page_size:0,
> compressed_page_size:0)
> Fragment 0:0
> [Error Id: eb6a5df8-fc59-409b-957a-59cb1079b5b8 on atsqa4-133.qa.lab:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at
> sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> drillbit.log
> {code}
> 2015-09-30 21:15:45,710 [29f3aefe-3209-a6e6-0418-500dac60a339:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
> 2015-09-30 21:15:45,712 [29f3aefe-3209-a6e6-0418-500dac60a339:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Time: 1ms total, 1.645381ms avg, 1ms max.
> 2015-09-30 21:15:45,712 [29f3aefe-3209-a6e6-0418-500dac60a339:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Earliest start: 1.332000 μs, Latest start: 1.332000 μs,
> Average start: 1.332000 μs .
> 2015-09-30 21:15:45,830 [29f3aefe-3209-a6e6-0418-500dac60a339:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29f3aefe-3209-a6e6-0418-500dac60a339:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2015-09-30 21:15:45,830 [29f3aefe-3209-a6e6-0418-500dac60a339:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 29f3aefe-3209-a6e6-0418-500dac60a339:0:0: State to report: RUNNING
> 2015-09-30 21:15:45,925 [29f3aefe-3209-a6e6-0418-500dac60a339:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29f3aefe-3209-a6e6-0418-500dac60a339:0:0: State change requested RUNNING -->
> FAILED
> 2015-09-30 21:15:45,930 [29f3aefe-3209-a6e6-0418-500dac60a339:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29f3aefe-3209-a6e6-0418-500dac60a339:0:0: State change requested FAILED -->
> FINISHED
> 2015-09-30 21:15:45,931 [29f3aefe-3209-a6e6-0418-500dac60a339:frag:0:0] ERROR
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: TProtocolException:
> Required field 'uncompressed_page_size' was not found in serialized data!
> Struct: PageHeader(type:null, uncompressed_page_size:0,
> compressed_page_size:0)
> Fragment 0:0
> [Error Id: eb6a5df8-fc59-409b-957a-59cb1079b5b8 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> TProtocolException: Required field 'uncompressed_page_size' was not found in
> serialized data! Struct: PageHeader(type:null, uncompressed_page_size:0,
> compressed_page_size:0)
> Fragment 0:0
> [Error Id: eb6a5df8-fc59-409b-957a-59cb1079b5b8 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
>
[jira] [Updated] (DRILL-2601) Print SQL query text along with query id in drillbit.log
[ https://issues.apache.org/jira/browse/DRILL-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-2601: Reviewer: Victoria Markman > Print SQL query text along with query id in drillbit.log > > > Key: DRILL-2601 > URL: https://issues.apache.org/jira/browse/DRILL-2601 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Reporter: Victoria Markman >Assignee: Aman Sinha > Fix For: 1.4.0 > > Attachments: > 0001-DRILL-2601-log-the-query-text-and-query-id-right-at-.patch > > > This is a request to print text of a query into drillbit.log in the default > non verbose output. It includes all the changes of a session level parameters > and anything else that might help reproducing issue on a customer site. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3802) Throw unsupported error for ROLLUP/GROUPING
[
https://issues.apache.org/jira/browse/DRILL-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3802:
Component/s: Query Planning & Optimization
> Throw unsupported error for ROLLUP/GROUPING
> ---
>
> Key: DRILL-3802
> URL: https://issues.apache.org/jira/browse/DRILL-3802
> Project: Apache Drill
> Issue Type: New Feature
> Components: Query Planning & Optimization
>Reporter: Victoria Markman
>Assignee: Jinfeng Ni
> Fix For: 1.3.0
>
>
> I believe that this is the cause of assertions in TPCDS #36, #67
> {code}
> SELECT Sum(ss_net_profit) / Sum(ss_ext_sales_price) AS
> gross_margin,
> i_category,
> i_class,
> Grouping(i_category) + Grouping(i_class) AS
> lochierarchy,
> Rank()
> OVER (
> partition BY Grouping(i_category)+Grouping(i_class), CASE
> WHEN Grouping(
> i_class) = 0 THEN i_category END
> ORDER BY Sum(ss_net_profit)/Sum(ss_ext_sales_price) ASC) AS
> rank_within_parent
> FROM store_sales,
> date_dim d1,
> item,
> store
> WHERE d1.d_year = 2000
> AND d1.d_date_sk = ss_sold_date_sk
> AND i_item_sk = ss_item_sk
> AND s_store_sk = ss_store_sk
> AND s_state IN ( 'TN', 'TN', 'TN', 'TN',
> 'TN', 'TN', 'TN', 'TN' )
> GROUP BY rollup( i_category, i_class )
> ORDER BY lochierarchy DESC,
> CASE
> WHEN lochierarchy = 0 THEN i_category
> END,
> rank_within_parent
> LIMIT 100;
> Error: SYSTEM ERROR: AssertionError: Internal error: invariant violated:
> conversion result not null
> [Error Id: 6afae7ce-c426-44f3-a600-aa34ab7632a1 on ucs-node5.perf.lab:31010]
> (state=,code=0)
> java.sql.SQLException: SYSTEM ERROR: AssertionError: Internal error:
> invariant violated: conversion result not null
> [Error Id: 6afae7ce-c426-44f3-a600-aa34ab7632a1 on ucs-node5.perf.lab:31010]
> at
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
> at
> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:290)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1359)
> at
> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:74)
> at
> net.hydromatic.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:404)
> at
> net.hydromatic.avatica.AvaticaStatement.executeQueryInternal(AvaticaStatement.java:351)
> at
> net.hydromatic.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:338)
> at net.hydromatic.avatica.AvaticaStatement.execute(AvaticaStatement.java:69)
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:86)
> at sqlline.Commands.execute(Commands.java:841)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.runCommands(SqlLine.java:1641)
> at sqlline.Commands.run(Commands.java:1304)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
> at sqlline.SqlLine.dispatch(SqlLine.java:734)
> at sqlline.SqlLine.initArgs(SqlLine.java:544)
> at sqlline.SqlLine.begin(SqlLine.java:587)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM
> ERROR: AssertionError: Internal error: invariant violated: conversion result
> not null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2419) UDF that returns string representation of expression type
[ https://issues.apache.org/jira/browse/DRILL-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-2419: Reviewer: Victoria Markman > UDF that returns string representation of expression type > - > > Key: DRILL-2419 > URL: https://issues.apache.org/jira/browse/DRILL-2419 > Project: Apache Drill > Issue Type: Improvement > Components: Functions - Drill >Reporter: Victoria Markman >Assignee: Steven Phillips > Fix For: 1.3.0 > > > Suggested name: typeof (credit goes to Aman) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4109) NPE in RecordIterator
[
https://issues.apache.org/jira/browse/DRILL-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4109:
Component/s: Execution - Relational Operators
> NPE in RecordIterator
> -
>
> Key: DRILL-4109
> URL: https://issues.apache.org/jira/browse/DRILL-4109
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac6c1b-9b33-3457-8bc8-9e2dff6ad438.sys.drill,
> 29b41f37-4803-d7ce-e05f-912d1f65da79.sys.drill, drillbit.log,
> drillbit.log.debug
>
>
> 4 node cluster
> 36GB of direct memory
> 4GB heap memory
> planner.memory.max_query_memory_per_node=2GB (default)
> planner.enable_hashjoin = false
> Spill directory has 6.4T of memory available:
> {noformat}
> [Tue Nov 17 18:23:18 /tmp/drill ] # df -H .
> Filesystem Size Used Avail Use% Mounted on
> localhost:/mapr 7.7T 1.4T 6.4T 18% /mapr
> {noformat}
> Run query below:
> framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> drillbit.log
> {code}
> 2015-11-18 02:22:12,639 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:9] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_9/operator_17/7
> 2015-11-18 02:22:12,770 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:5] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_5/operator_17/7
> 2015-11-18 02:22:13,345 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:17] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_17/operator_17/7
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_13/operator_16/1
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 34 batch groups.
> Current allocated memory: 2252186
> 2015-11-18 02:22:13,363 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested RUNNING -->
> FAILED
> 2015-11-18 02:22:13,370 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested FAILED -->
> FINISHED
> 2015-11-18 02:22:13,371 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> java.lang.NullPointerException: null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-2601) Print SQL query text along with query id in drillbit.log
[ https://issues.apache.org/jira/browse/DRILL-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman closed DRILL-2601. --- > Print SQL query text along with query id in drillbit.log > > > Key: DRILL-2601 > URL: https://issues.apache.org/jira/browse/DRILL-2601 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Reporter: Victoria Markman >Assignee: Aman Sinha > Fix For: 1.4.0 > > Attachments: > 0001-DRILL-2601-log-the-query-text-and-query-id-right-at-.patch > > > This is a request to print text of a query into drillbit.log in the default > non verbose output. It includes all the changes of a session level parameters > and anything else that might help reproducing issue on a customer site. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2601) Print SQL query text along with query id in drillbit.log
[
https://issues.apache.org/jira/browse/DRILL-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053113#comment-15053113
]
Victoria Markman commented on DRILL-2601:
-
Verified that following is printed in drillbit.log:
- use
- select from table
- select from sys.options
- query that throws syntax error
- ddl ( create view )
- alter session (alter session set `planner.enable_hashjoin` = false)
- alter system (alter system set `planner.enable_hashjoin` = false)
- show tables
Verified in 1.4.0
{code}
#Generated by Git-Commit-Id-Plugin
#Tue Dec 08 03:32:09 UTC 2015
git.commit.id.abbrev=b906811
git.commit.user.email=amit.ha...@gmail.com
git.commit.message.full=DRILL-4165 Add a precondition for size of merge join
record batch.\n
git.commit.id=b9068117177c3b47025f52c00f67938e0c3e4732
git.commit.message.short=DRILL-4165 Add a precondition for size of merge join
record batch.
git.commit.user.name=Amit Hadke
git.build.user.name=Unknown
git.commit.id.describe=0.9.0-518-gb906811-dirty
git.build.user.email=Unknown
git.branch=master
git.commit.time=08.12.2015 @ 01\:25\:34 UTC
git.build.time=08.12.2015 @ 03\:32\:09 UTC
git.remote.origin.url=https\://github.com/apache/drill
{code}
> Print SQL query text along with query id in drillbit.log
>
>
> Key: DRILL-2601
> URL: https://issues.apache.org/jira/browse/DRILL-2601
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Victoria Markman
>Assignee: Aman Sinha
> Fix For: 1.4.0
>
> Attachments:
> 0001-DRILL-2601-log-the-query-text-and-query-id-right-at-.patch
>
>
> This is a request to print text of a query into drillbit.log in the default
> non verbose output. It includes all the changes of a session level parameters
> and anything else that might help reproducing issue on a customer site.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4126) Adding HiveMetaStore caching when impersonation is enabled.
[ https://issues.apache.org/jira/browse/DRILL-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-4126: Fix Version/s: 1.4.0 > Adding HiveMetaStore caching when impersonation is enabled. > > > Key: DRILL-4126 > URL: https://issues.apache.org/jira/browse/DRILL-4126 > Project: Apache Drill > Issue Type: Bug >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.4.0 > > > Currently, HiveMetastore caching is used only when impersonation is disabled, > such that all the hivemetastore call goes through > NonCloseableHiveClientWithCaching [1]. However, if impersonation is enabled, > caching is not used for HiveMetastore access. > This could significantly increase the planning time when hive storage plugin > is enabled, or when running a query against INFORMATION_SCHEMA. Depending on > the # of databases/tables in Hive storage plugin, the planning time or > INFORMATION_SCHEMA query could become unacceptable. This becomes even worse > if the hive metastore is running on a different node from drillbit, making > the access of hivemetastore even slower. > We are seeing that it could takes 30~60 seconds for planning time, or > execution time for INFORMATION_SCHEMA query. The long planning or execution > time for INFORMATION_SCHEMA query prevents Drill from acting "interactively" > for such queries. > We should enable caching when impersonation is used. As long as the > authorizer verifies the user has the access to databases/tables, we should > get the data from caching. By doing that, we should see reduced number of api > call to HiveMetaStore. > [1] > https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java#L299 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4126) Adding HiveMetaStore caching when impersonation is enabled.
[ https://issues.apache.org/jira/browse/DRILL-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman updated DRILL-4126: Fix Version/s: (was: 1.4.0) 1.5.0 > Adding HiveMetaStore caching when impersonation is enabled. > > > Key: DRILL-4126 > URL: https://issues.apache.org/jira/browse/DRILL-4126 > Project: Apache Drill > Issue Type: Bug >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.5.0 > > > Currently, HiveMetastore caching is used only when impersonation is disabled, > such that all the hivemetastore call goes through > NonCloseableHiveClientWithCaching [1]. However, if impersonation is enabled, > caching is not used for HiveMetastore access. > This could significantly increase the planning time when hive storage plugin > is enabled, or when running a query against INFORMATION_SCHEMA. Depending on > the # of databases/tables in Hive storage plugin, the planning time or > INFORMATION_SCHEMA query could become unacceptable. This becomes even worse > if the hive metastore is running on a different node from drillbit, making > the access of hivemetastore even slower. > We are seeing that it could takes 30~60 seconds for planning time, or > execution time for INFORMATION_SCHEMA query. The long planning or execution > time for INFORMATION_SCHEMA query prevents Drill from acting "interactively" > for such queries. > We should enable caching when impersonation is used. As long as the > authorizer verifies the user has the access to databases/tables, we should > get the data from caching. By doing that, we should see reduced number of api > call to HiveMetaStore. > [1] > https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java#L299 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4109) NPE in RecordIterator
[
https://issues.apache.org/jira/browse/DRILL-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053604#comment-15053604
]
Victoria Markman commented on DRILL-4109:
-
Verified again with 1.4.0 with the fix for DRILL-4165
{code}
#Tue Dec 08 03:32:09 UTC 2015
git.commit.id.abbrev=b906811
git.commit.user.email=amit.ha...@gmail.com
git.commit.message.full=DRILL-4165 Add a precondition for size of merge join
record batch.\n
git.commit.id=b9068117177c3b47025f52c00f67938e0c3e4732
git.commit.message.short=DRILL-4165 Add a precondition for size of merge join
record batch.
git.commit.user.name=Amit Hadke
{code}
> NPE in RecordIterator
> -
>
> Key: DRILL-4109
> URL: https://issues.apache.org/jira/browse/DRILL-4109
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac6c1b-9b33-3457-8bc8-9e2dff6ad438.sys.drill,
> 29b41f37-4803-d7ce-e05f-912d1f65da79.sys.drill, drillbit.log,
> drillbit.log.debug
>
>
> 4 node cluster
> 36GB of direct memory
> 4GB heap memory
> planner.memory.max_query_memory_per_node=2GB (default)
> planner.enable_hashjoin = false
> Spill directory has 6.4T of memory available:
> {noformat}
> [Tue Nov 17 18:23:18 /tmp/drill ] # df -H .
> Filesystem Size Used Avail Use% Mounted on
> localhost:/mapr 7.7T 1.4T 6.4T 18% /mapr
> {noformat}
> Run query below:
> framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> drillbit.log
> {code}
> 2015-11-18 02:22:12,639 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:9] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_9/operator_17/7
> 2015-11-18 02:22:12,770 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:5] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_5/operator_17/7
> 2015-11-18 02:22:13,345 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:17] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_17/operator_17/7
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_13/operator_16/1
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 34 batch groups.
> Current allocated memory: 2252186
> 2015-11-18 02:22:13,363 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested RUNNING -->
> FAILED
> 2015-11-18 02:22:13,370 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested FAILED -->
> FINISHED
> 2015-11-18 02:22:13,371 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> java.lang.NullPointerException: null
> {code}
--
This message was sent by
[jira] [Closed] (DRILL-4109) NPE in RecordIterator
[
https://issues.apache.org/jira/browse/DRILL-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-4109.
---
> NPE in RecordIterator
> -
>
> Key: DRILL-4109
> URL: https://issues.apache.org/jira/browse/DRILL-4109
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac6c1b-9b33-3457-8bc8-9e2dff6ad438.sys.drill,
> 29b41f37-4803-d7ce-e05f-912d1f65da79.sys.drill, drillbit.log,
> drillbit.log.debug
>
>
> 4 node cluster
> 36GB of direct memory
> 4GB heap memory
> planner.memory.max_query_memory_per_node=2GB (default)
> planner.enable_hashjoin = false
> Spill directory has 6.4T of memory available:
> {noformat}
> [Tue Nov 17 18:23:18 /tmp/drill ] # df -H .
> Filesystem Size Used Avail Use% Mounted on
> localhost:/mapr 7.7T 1.4T 6.4T 18% /mapr
> {noformat}
> Run query below:
> framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> drillbit.log
> {code}
> 2015-11-18 02:22:12,639 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:9] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_9/operator_17/7
> 2015-11-18 02:22:12,770 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:5] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_5/operator_17/7
> 2015-11-18 02:22:13,345 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:17] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_17/operator_17/7
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/29b41f37-4803-d7ce-e05f-912d1f65da79/major_fragment_3/minor_fragment_13/operator_16/1
> 2015-11-18 02:22:13,346 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 34 batch groups.
> Current allocated memory: 2252186
> 2015-11-18 02:22:13,363 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested RUNNING -->
> FAILED
> 2015-11-18 02:22:13,370 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 29b41f37-4803-d7ce-e05f-912d1f65da79:3:13: State change requested FAILED -->
> FINISHED
> 2015-11-18 02:22:13,371 [29b41f37-4803-d7ce-e05f-912d1f65da79:frag:3:13]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> NullPointerException
> Fragment 3:13
> [Error Id: c5d67dcb-16aa-4951-89f5-599b4b4eb54d on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> java.lang.NullPointerException: null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-4125) Illegal argument exception during merge join
[
https://issues.apache.org/jira/browse/DRILL-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-4125.
---
> Illegal argument exception during merge join
> -
>
> Key: DRILL-4125
> URL: https://issues.apache.org/jira/browse/DRILL-4125
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac59f2-5d92-7378-bf81-e844a300efd7.sys.drill,
> drillbit.log
>
>
> Same setup as in DRILL-4109
> Query: framework/resources/Advanced/tpcds/tpcds_sf100/original/query93.sql
> Excerpt from drillbit.log
> {code}
> 2015-11-23 23:50:44,071 [29ac59f2-5d92-7378-bf81-e844a300efd7:frag:5:74]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalArgumentException
> Fragment 5:74
> [Error Id: 1ca9758d-1864-4940-9efa-b8906d4f9b52 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException
> Fragment 5:74
> [Error Id: 1ca9758d-1864-4940-9efa-b8906d4f9b52 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: null
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> ~[guava-14.0.1.jar:na]
> at
> org.apache.drill.exec.record.RecordIterator.getCurrentPosition(RecordIterator.java:242)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.JoinWorkerGen8348.doJoin(JoinTemplate.java:63)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:206)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:276)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apa
[jira] [Commented] (DRILL-4125) Illegal argument exception during merge join
[
https://issues.apache.org/jira/browse/DRILL-4125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053610#comment-15053610
]
Victoria Markman commented on DRILL-4125:
-
Verified again with:
{code}
#Tue Dec 08 03:32:09 UTC 2015
git.commit.id.abbrev=b906811
git.commit.user.email=amit.ha...@gmail.com
git.commit.message.full=DRILL-4165 Add a precondition for size of merge join
record batch.\n
git.commit.id=b9068117177c3b47025f52c00f67938e0c3e4732
git.commit.message.short=DRILL-4165 Add a precondition for size of merge join
record batch.
git.commit.user.name=Amit Hadke
{code}
> Illegal argument exception during merge join
> -
>
> Key: DRILL-4125
> URL: https://issues.apache.org/jira/browse/DRILL-4125
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Victoria Markman
>Assignee: amit hadke
>Priority: Blocker
> Fix For: 1.4.0
>
> Attachments: 29ac59f2-5d92-7378-bf81-e844a300efd7.sys.drill,
> drillbit.log
>
>
> Same setup as in DRILL-4109
> Query: framework/resources/Advanced/tpcds/tpcds_sf100/original/query93.sql
> Excerpt from drillbit.log
> {code}
> 2015-11-23 23:50:44,071 [29ac59f2-5d92-7378-bf81-e844a300efd7:frag:5:74]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalArgumentException
> Fragment 5:74
> [Error Id: 1ca9758d-1864-4940-9efa-b8906d4f9b52 on atsqa4-133.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalArgumentException
> Fragment 5:74
> [Error Id: 1ca9758d-1864-4940-9efa-b8906d4f9b52 on atsqa4-133.qa.lab:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
> ~[drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:321)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
> [drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: null
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> ~[guava-14.0.1.jar:na]
> at
> org.apache.drill.exec.record.RecordIterator.getCurrentPosition(RecordIterator.java:242)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.JoinWorkerGen8348.doJoin(JoinTemplate.java:63)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:206)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:276)
> ~[drill-java-exec-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
> at
> org
[jira] [Updated] (DRILL-3543) Add stats for external sort to a query profile
[
https://issues.apache.org/jira/browse/DRILL-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3543:
Labels: docum usability (was: usability)
> Add stats for external sort to a query profile
> --
>
> Key: DRILL-3543
> URL: https://issues.apache.org/jira/browse/DRILL-3543
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: docum, usability
> Fix For: 1.4.0
>
>
> The only indication if sort spilled to disk today is info from the
> drillbit.log.
> It would be great if this information was displayed in the query profile.
> {code}
> 015-07-22 23:47:29,907 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/92
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 7 batch groups.
> Current allocated memory: 11566787
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3543) Add stats for external sort to a query profile
[
https://issues.apache.org/jira/browse/DRILL-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3543:
Labels: documentation usability (was: docum usability)
> Add stats for external sort to a query profile
> --
>
> Key: DRILL-3543
> URL: https://issues.apache.org/jira/browse/DRILL-3543
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: documentation, usability
> Fix For: 1.4.0
>
>
> The only indication if sort spilled to disk today is info from the
> drillbit.log.
> It would be great if this information was displayed in the query profile.
> {code}
> 015-07-22 23:47:29,907 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/92
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 7 batch groups.
> Current allocated memory: 11566787
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3543) Add stats for external sort to a query profile
[
https://issues.apache.org/jira/browse/DRILL-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-3543:
Attachment: Screen Shot 2015-12-11 at 2.19.37 PM.png
> Add stats for external sort to a query profile
> --
>
> Key: DRILL-3543
> URL: https://issues.apache.org/jira/browse/DRILL-3543
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: documentation, usability
> Fix For: 1.4.0
>
> Attachments: Screen Shot 2015-12-11 at 2.19.37 PM.png
>
>
> The only indication if sort spilled to disk today is info from the
> drillbit.log.
> It would be great if this information was displayed in the query profile.
> {code}
> 015-07-22 23:47:29,907 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/92
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 7 batch groups.
> Current allocated memory: 11566787
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3543) Add stats for external sort to a query profile
[
https://issues.apache.org/jira/browse/DRILL-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053679#comment-15053679
]
Victoria Markman commented on DRILL-3543:
-
We need to document the fact that after upgrade to 1.4.0 prior profiles that
contained external sort operator in them are not going to display stats
correctly. All the fields will be empty.
Screen shot attached.
> Add stats for external sort to a query profile
> --
>
> Key: DRILL-3543
> URL: https://issues.apache.org/jira/browse/DRILL-3543
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
>Affects Versions: 1.1.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Critical
> Labels: documentation, usability
> Fix For: 1.4.0
>
> Attachments: Screen Shot 2015-12-11 at 2.19.37 PM.png
>
>
> The only indication if sort spilled to disk today is info from the
> drillbit.log.
> It would be great if this information was displayed in the query profile.
> {code}
> 015-07-22 23:47:29,907 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/92
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Merging and spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] INFO
> o.a.d.e.p.i.xsort.ExternalSortBatch - Completed spilling to
> /tmp/drill/spill/2a4fd46e-f8c3-6b96-b165-b665a41be311/major_fragment_0/minor_fragment_0/operator_7/93
> 2015-07-22 23:47:29,919 [2a4fd46e-f8c3-6b96-b165-b665a41be311:frag:0:0] WARN
> o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 7 batch groups.
> Current allocated memory: 11566787
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
Victoria Markman created DRILL-4190:
---
Summary: TPCDS queries are running out of memory when hash join is
disabled
Key: DRILL-4190
URL: https://issues.apache.org/jira/browse/DRILL-4190
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.3.0, 1.4.0, 1.5.0
Reporter: Victoria Markman
Priority: Blocker
TPCDS queries with the latest 1.4.0 release when hash join is disabled:
22 queries fail with out of memory
2 wrong results (I did not validate the nature of wrong result yet)
Only query97.sql is a legitimate failure: we don't support full outer join with
the merge join.
It is important to understand what has changed between 1.2.0 and 1.4.0 that
made these tests not runnable with the same configuration.
Same tests with the same drill configuration pass in 1.2.0 release.
(I hope I did not make a mistake somewhere in my cluster setup :))
*mapr-drill-1.4.0.201512071931-1.noarch.rpm*
{code}
Execution Failures:
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
Verification Failures
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
Timeout Failures
Passing tests: 3
Execution Failures: 22
VerificationFailures: 2
Timeouts: 0
Canceled: 0
{code}
*mapr-drill-1.2.0.201510190924-1.noarch.rpm*
{code}
PASS (1.543 min)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
(connection: 1681915178)
PASS (29.36 s)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
(connection: 1681915178)
PASS (3.311 min)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
(connection: 1681915178)
PASS (1.447 min)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
(connection: 1681915178)
PASS (34.53 s)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query76.sql
(connection: 1681915178)
PASS (47.13 s)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
(connection: 1681915178)
PASS (1.151 min)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
(connection: 1681915178)
PASS (32.29 s)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
(connection: 1681915178)
PASS (1.939 min)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
(connection: 1681915178)
PASS (19.26 s)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
(connection: 1681915178)
PASS (1.243 min)
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/ori
[jira] [Updated] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4190:
Description:
TPCDS queries with the latest 1.4.0 release when hash join is disabled:
22 queries fail with out of memory
2 wrong results (I did not validate the nature of wrong result yet)
Only query97.sql is a legitimate failure: we don't support full outer join with
the merge join.
It is important to understand what has changed between 1.2.0 and 1.4.0 that
made these tests not runnable with the same configuration.
Same tests with the same drill configuration pass in 1.2.0 release.
(I hope I did not make a mistake somewhere in my cluster setup :))
{code}
0: jdbc:drill:schema=dfs> select * from sys.version;
+-+---+-++--++
| version | commit_id |
commit_message|commit_time
| build_email | build_time |
+-+---+-++--++
| 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165 Add
a precondition for size of merge join record batch. | 08.12.2015 @ 01:25:34
UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
+-+---+-++--++
1 row selected (2.211 seconds)
Execution Failures:
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
Verification Failures
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
/root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
Timeout Failures
Passing tests: 3
Execution Failures: 22
VerificationFailures: 2
Timeouts: 0
Canceled: 0
{code}
{code}
0: jdbc:drill:schema=dfs> select * from sys.version;
+---++-+-++
| commit_id | commit_message | commit_time | build_email | build_time |
+---++-+-++
| f1100a79b4e4fbb1b58b35b0230edff137588777 | DRILL-3947: Use setSafe() for
date, time, timestamp types while populating pruning vector (other types were
already using setSafe). | 19.10.2015 @ 16:02:00 UTC | Unknown | 19.10.2015 @
16:25:21 UTC |
+---++-+-++
1 row selected (2.79 seconds)
PASS (1.543 min)
/root/drill-tests-new/frame
[jira] [Commented] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053850#comment-15053850
]
Victoria Markman commented on DRILL-4190:
-
I'm thinking that we might have two different bugs here.
> TPCDS queries are running out of memory when hash join is disabled
> --
>
> Key: DRILL-4190
> URL: https://issues.apache.org/jira/browse/DRILL-4190
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Victoria Markman
>Priority: Blocker
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join
> with the merge join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that
> made these tests not runnable with the same configuration.
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
> Timeout Failures
>
> Passing tests: 3
> Execution Failures: 22
> VerificationFailures: 2
> Timeouts: 0
> Canceled: 0
[jira] [Updated] (DRILL-4191) Last value function returns incorrect results.
[
https://issues.apache.org/jira/browse/DRILL-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4191:
Priority: Critical (was: Major)
> Last value function returns incorrect results.
> --
>
> Key: DRILL-4191
> URL: https://issues.apache.org/jira/browse/DRILL-4191
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.4.0
> Environment: 4 node cluster on CentOS
>Reporter: Khurram Faraaz
>Priority: Critical
>
> Last value function returns incorrect results.
> {noformat}
> [root@centos-01 ~]# cat MD627.csv
> 1,2015-01-01
> 1,2015-01-02
> 1,2015-12-12
> git commit id : b9068117
> create table md_627 as select cast(columns[0] as int) c1, cast(columns[1] as
> date) c2 from `MD627.csv`;
> [root@centos-01 parquet-tools]# ./parquet-schema ../md627/0_0_0.parquet
> message root {
> optional int32 c1;
> optional int32 c2 (DATE);
> }
> 0: jdbc:drill:schema=dfs.tmp> select * from md_627;
> +-+-+
> | c1 | c2 |
> +-+-+
> | 1 | 2015-01-01 |
> | 1 | 2015-01-02 |
> | 1 | 2015-12-12 |
> +-+-+
> 3 rows selected (0.265 seconds)
> 0: jdbc:drill:schema=dfs.tmp> select c1, last_value(c2) over(partition by c1
> order by c2) lstval from md_627;
> +-+-+
> | c1 | lstval|
> +-+-+
> | 1 | 2015-01-01 |
> | 1 | 2015-01-02 |
> | 1 | 2015-12-12 |
> +-+-+
> 3 rows selected (0.405 seconds)
> 0: jdbc:drill:schema=dfs.tmp> explain plan for select c1, last_value(c2)
> over(partition by c1 order by c2) lstval from md_627;
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(c1=[$0], lstval=[$1])
> 00-02Project(c1=[$0], lstval=[$1])
> 00-03 Project(c1=[$0], $1=[$2])
> 00-04Window(window#0=[window(partition {0} order by [1] range
> between UNBOUNDED PRECEDING and CURRENT ROW aggs [LAST_VALUE($1)])])
> 00-05 SelectionVectorRemover
> 00-06Sort(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> 00-07 Project(c1=[$1], c2=[$0])
> 00-08Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath [path=maprfs:///tmp/md_627]],
> selectionRoot=maprfs:/tmp/md_627, numFiles=1, usedMetadataFile=false,
> columns=[`c1`, `c2`]]])
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4196) some TPCDS queries return wrong result when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4196:
Attachment: query40.tar
query52.tar
> some TPCDS queries return wrong result when hash join is disabled
> -
>
> Key: DRILL-4196
> URL: https://issues.apache.org/jira/browse/DRILL-4196
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Victoria Markman
> Attachments: query40.tar, query52.tar
>
>
> With hash join disabled query52.sql and query40.sql returned incorrect result
> with 1.4.0 :
> {noformat}
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.13 seconds)
> {noformat}
> Setup and options are the same as in DRILL-4190
> See attached queries (.sql), expected result (.e_tsv) and actual output (.out)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056406#comment-15056406
]
Victoria Markman commented on DRILL-4190:
-
I filed a separate bug (DRILL-4196) for the wrong result. This bug only
addresses running out of memory regression.
> TPCDS queries are running out of memory when hash join is disabled
> --
>
> Key: DRILL-4190
> URL: https://issues.apache.org/jira/browse/DRILL-4190
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Victoria Markman
>Priority: Blocker
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join
> with the merge join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that
> made these tests not runnable with the same configuration.
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
> Timeout Failures
>
> Passing tests: 3
> Execution Failur
[jira] [Created] (DRILL-4196) some TPCDS queries return wrong result when hash join is disabled
Victoria Markman created DRILL-4196:
---
Summary: some TPCDS queries return wrong result when hash join is
disabled
Key: DRILL-4196
URL: https://issues.apache.org/jira/browse/DRILL-4196
Project: Apache Drill
Issue Type: Bug
Reporter: Victoria Markman
With hash join disabled query52.sql and query40.sql returned incorrect result
with 1.4.0 :
{noformat}
+-+---+-++--++
| version | commit_id |
commit_message|commit_time
| build_email | build_time |
+-+---+-++--++
| 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165 Add
a precondition for size of merge join record batch. | 08.12.2015 @ 01:25:34
UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
+-+---+-++--++
1 row selected (2.13 seconds)
{noformat}
Setup and options are the same as in DRILL-4190
See attached queries (.sql), expected result (.e_tsv) and actual output (.out)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4190:
Attachment: 2990f5f8-ec64-1223-c1d8-97dd7e601cee.sys.drill
query3.sql
> TPCDS queries are running out of memory when hash join is disabled
> --
>
> Key: DRILL-4190
> URL: https://issues.apache.org/jira/browse/DRILL-4190
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Victoria Markman
>Priority: Blocker
> Attachments: 2990f5f8-ec64-1223-c1d8-97dd7e601cee.sys.drill,
> query3.sql
>
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join
> with the merge join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that
> made these tests not runnable with the same configuration.
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
> Timeout Failures
>
> Passing tests: 3
> Execution Fail
[jira] [Updated] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman updated DRILL-4190:
Attachment: exception.log
> TPCDS queries are running out of memory when hash join is disabled
> --
>
> Key: DRILL-4190
> URL: https://issues.apache.org/jira/browse/DRILL-4190
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Victoria Markman
>Priority: Blocker
> Attachments: 2990f5f8-ec64-1223-c1d8-97dd7e601cee.sys.drill,
> exception.log, query3.sql
>
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join
> with the merge join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that
> made these tests not runnable with the same configuration.
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
> Timeout Failures
>
> Passing tests: 3
> Execution Failures: 22
> VerificationFailures: 2
> Timeouts
[jira] [Commented] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056451#comment-15056451
]
Victoria Markman commented on DRILL-4190:
-
It looks like OOM is happening in major fragment 5 in external sort. See
exception.log attached. However, parquet row group scan utilization looks very
suspicious to me, is it too high ?
> TPCDS queries are running out of memory when hash join is disabled
> --
>
> Key: DRILL-4190
> URL: https://issues.apache.org/jira/browse/DRILL-4190
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Victoria Markman
>Priority: Blocker
> Attachments: 2990f5f8-ec64-1223-c1d8-97dd7e601cee.sys.drill,
> exception.log, query3.sql
>
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join
> with the merge join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that
> made these tests not runnable with the same configuration.
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.s
[jira] [Commented] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056824#comment-15056824
]
Victoria Markman commented on DRILL-2967:
-
This bug is fixed, verified with:
{code}
#Tue Dec 08 03:32:09 UTC 2015
git.commit.id.abbrev=b906811
git.commit.user.email=amit.ha...@gmail.com
git.commit.message.full=DRILL-4165 Add a precondition for size of merge join
record batch.\n
git.commit.id=b9068117177c3b47025f52c00f67938e0c3e4732
{code}
Verification steps: ran Functional/subqueries/not_in test suite with options
below.
Options test suite was executed with:
{code}
0: jdbc:drill:schema=dfs> select * from sys.options where status like
'%CHANGED%';
+---+--+-+--+--+-+---++
| name| kind | type | status |
num_val| string_val | bool_val | float_val |
+---+--+-+--+--+-+---++
| planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED |
null | null| true | null |
| planner.enable_hashjoin | BOOLEAN | SYSTEM | CHANGED |
null | null| false | null |
| planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED |
10737418240 | null| null | null |
| planner.slice_target | LONG | SYSTEM | CHANGED | 1
| null| null | null |
+---+--+-+--+--+-+---++
4 rows selected (0.516 seconds)
{code}
Attached are the details of the run. 12 tests used to fail and they were caused
by this bug.
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
> Fix For: 1.3.0
>
> Attachments: j2_j6_tables.tar, t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2 where b2 not in (select b1 from
> t1);
> ++++
> | a1 | b1 | c1 |
> ++++
> Query failed: SYSTEM ERROR: Join only supports implicit casts between 1.
> Num
[jira] [Closed] (DRILL-2967) Incompatible types error reported in a "not in" query with compatible data types
[
https://issues.apache.org/jira/browse/DRILL-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-2967.
---
> Incompatible types error reported in a "not in" query with compatible data
> types
> -
>
> Key: DRILL-2967
> URL: https://issues.apache.org/jira/browse/DRILL-2967
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
> Fix For: 1.3.0
>
> Attachments: j2_j6_tables.tar, t1.parquet, t2.parquet
>
>
> Two tables, parquet files (attached in the bug):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1;
> ++++
> | a1 | b1 | c1 |
> ++++
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | null | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | null | h | 2015-01-08 |
> | 9 | i | null |
> | 10 | j | 2015-01-10 |
> ++++
> 10 rows selected (0.119 seconds)
> 0: jdbc:drill:schema=dfs> select * from t2;
> ++++
> | a2 | b2 | c2 |
> ++++
> | 0 | zzz| 2014-12-31 |
> | 1 | a | 2015-01-01 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 2 | b | 2015-01-02 |
> | 3 | c | 2015-01-03 |
> | 4 | d | 2015-01-04 |
> | 5 | e | 2015-01-05 |
> | 6 | f | 2015-01-06 |
> | 7 | g | 2015-01-07 |
> | 7 | g | 2015-01-07 |
> | 8 | h | 2015-01-08 |
> | 9 | i | 2015-01-09 |
> ++++
> 13 rows selected (0.116 seconds)
> {code}
> Disable hash join and set slice_target = 1:
> alter session set `planner.enable_hashjoin` = false;
> alter session set `planner.slice_target` = 1;
> Correct result:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1 where b1 not in (select b2 from
> t2);
> ++++
> | a1 | b1 | c1 |
> ++++
> | 10 | j | 2015-01-10 |
> ++++
> 1 row selected (0.625 seconds)
> {code}
> Swap tables and you get an error:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t2 where b2 not in (select b1 from
> t1);
> ++++
> | a1 | b1 | c1 |
> ++++
> Query failed: SYSTEM ERROR: Join only supports implicit casts between 1.
> Numeric data
> 2. Varchar, Varbinary data Left type: INT, Right type: VARCHAR. Add explicit
> casts to avoid this error
> Fragment 1:0
> [1a83aa50-39aa-452c-91dd-970bf4a8f03d on atsqa4-133.qa.lab:31010]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing
> query.
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> at sqlline.SqlLine.print(SqlLine.java:1809)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
> at sqlline.SqlLine.dispatch(SqlLine.java:889)
> at sqlline.SqlLine.begin(SqlLine.java:763)
> at sqlline.SqlLine.start(SqlLine.java:498)
> at sqlline.SqlLine.main(SqlLine.java:460)
> {code}
> Explain plan for the query with an error:
> {code}
> 0: jdbc:drill:schema=dfs> explain plan for select * from t2 where b2 not in
> (select b1 from t1);
> +++
> | text | json |
> +++
> | 00-00 Screen
> 00-01 Project(*=[$0])
> 00-02 UnionExchange
> 01-01 Project(T27¦¦*=[$0])
> 01-02 SelectionVectorRemover
> 01-03 Filter(condition=[NOT(CASE(=($2, 0), false, IS NOT NULL($6), true, IS
> NULL($4), null, <($3, $2), null, false))])
> 01-04 MergeJoin(condition=[=($4, $5)], joinType=[left])
> 01-06 SelectionVectorRemover
> 01-08 Sort(sort0=[$4], dir0=[ASC])
> 01-10 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4])
> 01-12 HashToRandomExchange(dist0=[[$4]])
> 02-01 UnorderedMuxExchange
> 04-01 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2], $f1=[$3], b20=[$4],
> E_X_P_R_H_A_S_H_F_I_E_L_D=[castInt(hash64AsDouble($4))])
> 04-02 Project(T27¦¦*=[$0], b2=[$1], $f0=[$2
[jira] [Commented] (DRILL-4190) TPCDS queries are running out of memory when hash join is disabled
[
https://issues.apache.org/jira/browse/DRILL-4190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057059#comment-15057059
]
Victoria Markman commented on DRILL-4190:
-
git.commit.id=b9068117177c3b47025f52c00f67938e0c3e4732
> TPCDS queries are running out of memory when hash join is disabled
> --
>
> Key: DRILL-4190
> URL: https://issues.apache.org/jira/browse/DRILL-4190
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0, 1.4.0, 1.5.0
>Reporter: Victoria Markman
>Assignee: Deneche A. Hakim
>Priority: Blocker
> Attachments: 2990f5f8-ec64-1223-c1d8-97dd7e601cee.sys.drill,
> exception.log, query3.sql
>
>
> TPCDS queries with the latest 1.4.0 release when hash join is disabled:
> 22 queries fail with out of memory
> 2 wrong results (I did not validate the nature of wrong result yet)
> Only query97.sql is a legitimate failure: we don't support full outer join
> with the merge join.
> It is important to understand what has changed between 1.2.0 and 1.4.0 that
> made these tests not runnable with the same configuration.
> Same tests with the same drill configuration pass in 1.2.0 release.
> (I hope I did not make a mistake somewhere in my cluster setup :))
> {code}
> 0: jdbc:drill:schema=dfs> select * from sys.version;
> +-+---+-++--++
> | version | commit_id |
> commit_message|commit_time
> | build_email | build_time |
> +-+---+-++--++
> | 1.4.0-SNAPSHOT | b9068117177c3b47025f52c00f67938e0c3e4732 | DRILL-4165
> Add a precondition for size of merge join record batch. | 08.12.2015 @
> 01:25:34 UTC | Unknown | 08.12.2015 @ 03:36:25 UTC |
> +-+---+-++--++
> 1 row selected (2.211 seconds)
> Execution Failures:
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query50.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query33.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query74.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query68.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query34.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query21.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query46.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query91.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query59.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query3.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query66.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query84.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query97.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query19.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query96.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query43.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query15.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query2.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query60.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query79.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query73.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query45.sql
> Verification Failures
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query52.sql
> /root/drill-tests-new/framework/resources/Advanced/tpcds/tpcds_sf100/original/query40.sql
> Timeout Failures
> --
[jira] [Closed] (DRILL-4165) IllegalStateException in MergeJoin for a query against TPC-DS data
[
https://issues.apache.org/jira/browse/DRILL-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victoria Markman closed DRILL-4165.
---
> IllegalStateException in MergeJoin for a query against TPC-DS data
> --
>
> Key: DRILL-4165
> URL: https://issues.apache.org/jira/browse/DRILL-4165
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: amit hadke
> Fix For: 1.4.0
>
>
> I am seeing the following on the 1.4.0 branch.
> {noformat}
> 0: jdbc:drill:zk=local> alter session set `planner.enable_hashjoin` = false;
> ..
> 0: jdbc:drill:zk=local> select count(*) from dfs.`tpcds/store_sales` ss1,
> dfs.`tpcds/store_sales` ss2 where ss1.ss_customer_sk = ss2.ss_customer_sk and
> ss1.ss_store_sk = 1 and ss2.ss_store_sk = 2;
> Error: SYSTEM ERROR: IllegalStateException: Incoming batch [#55,
> MergeJoinBatch] has size 1984616, which is beyond the limit of 65536
> Fragment 0:0
> [Error Id: 18bf00fe-52d7-4d84-97ec-b04a035afb4e on 192.168.1.103:31010]
> (java.lang.IllegalStateException) Incoming batch [#55, MergeJoinBatch] has
> size 1984616, which is beyond the limit of 65536
>
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():305
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():132
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4165) IllegalStateException in MergeJoin for a query against TPC-DS data
[
https://issues.apache.org/jira/browse/DRILL-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058621#comment-15058621
]
Victoria Markman commented on DRILL-4165:
-
Verified fixed in:
{code}
#Generated by Git-Commit-Id-Plugin
#Tue Dec 08 03:32:09 UTC 2015
git.commit.id.abbrev=b906811
git.commit.user.email=amit.ha...@gmail.com
git.commit.message.full=DRILL-4165 Add a precondition for size of merge join
record batch.\n
git.commit.id=b9068117177c3b47025f52c00f67938e0c3e4732
{code}
Test added under: Functional/tpcds/variants/parquet/drill-4165.sql
> IllegalStateException in MergeJoin for a query against TPC-DS data
> --
>
> Key: DRILL-4165
> URL: https://issues.apache.org/jira/browse/DRILL-4165
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.4.0
>Reporter: Aman Sinha
>Assignee: amit hadke
> Fix For: 1.4.0
>
>
> I am seeing the following on the 1.4.0 branch.
> {noformat}
> 0: jdbc:drill:zk=local> alter session set `planner.enable_hashjoin` = false;
> ..
> 0: jdbc:drill:zk=local> select count(*) from dfs.`tpcds/store_sales` ss1,
> dfs.`tpcds/store_sales` ss2 where ss1.ss_customer_sk = ss2.ss_customer_sk and
> ss1.ss_store_sk = 1 and ss2.ss_store_sk = 2;
> Error: SYSTEM ERROR: IllegalStateException: Incoming batch [#55,
> MergeJoinBatch] has size 1984616, which is beyond the limit of 65536
> Fragment 0:0
> [Error Id: 18bf00fe-52d7-4d84-97ec-b04a035afb4e on 192.168.1.103:31010]
> (java.lang.IllegalStateException) Incoming batch [#55, MergeJoinBatch] has
> size 1984616, which is beyond the limit of 65536
>
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():305
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
>
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():132
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
