[jira] [Updated] (DRILL-2422) Querying views having the same name as their datasource fails
[
https://issues.apache.org/jira/browse/DRILL-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated DRILL-2422:
---
Attachment: DRILL-2422-1.patch
Review Board Link: https://reviews.apache.org/r/33220/
Querying views having the same name as their datasource fails
-
Key: DRILL-2422
URL: https://issues.apache.org/jira/browse/DRILL-2422
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.8.0
Reporter: Abhishek Girish
Assignee: Venki Korukanti
Priority: Critical
Fix For: 0.9.0
Attachments: DRILL-2422-1.patch, drillbit_views.log
Git.Commit.ID: f658a3c (Mar 9 2015)
*Convert a simply text file to Parquet:*
{code:sql}
select * from `abc.dat` limit 1;
++
| columns |
++
| [hi,abc] |
++
1 row selected (0.574 seconds)
create table abc_1 as select cast(columns[0] as varchar(10)) as a from
`abc.dat`;
++---+
| Fragment | Number of records written |
++---+
| 0_0| 1 |
++---+
1 row selected (0.626 seconds)
select * from abc_1 limit 1;
++
| a |
++
| hi |
++
1 row selected (0.135 seconds)
{code}
*Create a view on top of the Parquet table:*
{code:sql}
create view abc_1 as select cast(a as varchar(10)) as b from abc_1;
+++
| ok | summary |
+++
| true | View 'abc_1' created successfully in 'dfs.tmp' schema |
+++
1 row selected (0.072 seconds)
select * from abc_1 limit 1;
Query failed: SqlValidatorException: Column 'a' not found in any table
Error: exception while executing query: Failure while executing query.
(state=,code=0)
OR
create view abc_1 as select cast(a as varchar(10)) as b from
dfs.`/tmp/abc_1`;
+++
| ok | summary |
+++
| true | View 'abc_1' created successfully in 'dfs.tmp' schema |
+++
1 row selected (0.047 seconds)
select * from abc_1 limit 1;
Query failed: SqlValidatorException: Column 'a' not found in any table
Error: exception while executing query: Failure while executing query.
(state=,code=0)
OR
create view abc_1 as select a from dfs.`/tmp/abc_1`;
+++
| ok | summary |
+++
| true | View 'abc_1' created successfully in 'dfs.tmp' schema |
+++
1 row selected (0.046 seconds)
select * from abc_1 limit 1;
+--+
| |
+--+
+--+
No rows selected (11.856 seconds)
{code}
In the third case, queries do not return and need to be cancelled.
For example:
{code:sql}
use dfs.tpcds1;
+--+
| |
+--+
+--+
No rows selected (2.854 seconds)
{code}
Sqlline needs to be suspended, killed and restarted to be able to query
again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2606) Casting a column from hbase to boolean after a join with parquet makes DRILL unresponsive
[
https://issues.apache.org/jira/browse/DRILL-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496294#comment-14496294
]
Venki Korukanti commented on DRILL-2606:
bq.1. Could you add a more general test case as well where we are casting to a
boolean type with exchanges in the plan, this would also test the
deserialization logic.
patch v2 adds one more test case
bq. 2. In the castBit (cast from varchar to boolean) function should we use
equalsIgnoreCase?
This seems like a bug in cast function implementation. Will log a separate JIRA.
Casting a column from hbase to boolean after a join with parquet makes DRILL
unresponsive
-
Key: DRILL-2606
URL: https://issues.apache.org/jira/browse/DRILL-2606
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 0.8.0
Reporter: Rahul Challapalli
Assignee: Venki Korukanti
Priority: Critical
Fix For: 0.9.0
Attachments: DRILL-2606-1.patch, DRILL-2606-2.patch, dataload.hql,
error.log, fewtypes_null.parquet, fewtypes_null_hbase.tbl
Small scale repro without HBase:
{code}
alter session set `planner.slice_target`=1;
select cast(cast(bool_col as varchar(100)) as boolean) from
dfs.`/Users/hadoop/Downloads/fewtypes_null.parquet` order by int_col;
{code}
git.commit.id.abbrev=4d398ed
The below query never returns and causes Sqlline to hang. DRILL also becomes
unresponsive after executing this query. So I have to restart drill for
sqlline to even come up.
{code}
select cast(cast(o.types.bool_col as varchar(100)) as boolean) bool_col
from dfs.`cross-sources`.`fewtypes_null.parquet` p
inner join hbase.fewtypes_null o
on p.int_col = cast(cast(o.types.int_col as varchar(100)) as int)
{code}
This issue is similar to DRILL-2605. However in this case I did not use a
view and some this causes drill/sqlline to become unresponsive where in
DRILL-2605 we just get back an error.
I attached the data files, data loading script, and the error log. Let me
know if you need anything else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2292) CTAS broken when we have repeated maps
[
https://issues.apache.org/jira/browse/DRILL-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-2292:
Attachment: DRILL-2292.1.patch.txt
- fixed {{RepeatedMapReaderImpl.setPosition()}} to handle when the index is
negative or *NO_VALUES*
- added unit test {{TestRepeatedReaders.testReadNestedRepeatedMap()}}
CTAS broken when we have repeated maps
--
Key: DRILL-2292
URL: https://issues.apache.org/jira/browse/DRILL-2292
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Reporter: Rahul Challapalli
Assignee: Deneche A. Hakim
Priority: Blocker
Fix For: 0.9.0
Attachments: DRILL-2292.1.patch.txt
git.commit.id.abbrev=6676f2d
Data Set :
{code}
{
id : 1,
map:{rm: [
{mapid:m1,mapvalue:{col1:1,col2:[0,1,2,3,4,5]},rptd: [{ a:
foo},{b:boo}]},
{mapid:m2,mapvalue:{col1:0,col2:[]},rptd: [{ a:
bar},{c:1},{d:4.5}]}
]}
}
{code}
The below CTAS statement fails though the underlying query succeeds
{code}
create table rep_map as select d.map from `temp.json` d;
Query failed: Query stopped., index: -4, length: 4 (expected: range(0,
16384)) [ d76e3f74-7e2c-406f-a7fd-5efc68227e75 on qa-node190.qa.lab:31010 ]
{code}
Error from the logs :
{code}
Caused by: java.lang.IndexOutOfBoundsException: index: -4, length: 4
(expected: range(0, 16384))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at
org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:309)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:451)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.reader(RepeatedMapReaderImpl.java:60)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter$RepeatedMapParquetConverter.init(ParquetRecordWriter.java:269)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewRepeatedMapConverter(ParquetRecordWriter.java:259)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:116)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.init(ParquetRecordWriter.java:240)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewMapConverter(ParquetRecordWriter.java:230)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:114)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.initFieldWriters(EventBasedRecordWriter.java:73)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
... 19 common frames omitted
2015-02-24 19:05:56,898 [2b13346a-d929-d659-c074-354c9f5d9d4f:frag:0:0] ERROR
o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
7c8fa189-8f0a-4763-a266-fd68a11bbb66: Query stopped.
java.io.IOException: Failed to initialize FieldWriters.
at
org.apache.drill.exec.store.EventBasedRecordWriter.initFieldWriters(EventBasedRecordWriter.java:78)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.init(EventBasedRecordWriter.java:46)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:176)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:113)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
[jira] [Updated] (DRILL-2292) CTAS broken when we have repeated maps
[
https://issues.apache.org/jira/browse/DRILL-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-2292:
Assignee: Steven Phillips (was: Deneche A. Hakim)
CTAS broken when we have repeated maps
--
Key: DRILL-2292
URL: https://issues.apache.org/jira/browse/DRILL-2292
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Reporter: Rahul Challapalli
Assignee: Steven Phillips
Priority: Blocker
Fix For: 0.9.0
Attachments: DRILL-2292.1.patch.txt
git.commit.id.abbrev=6676f2d
Data Set :
{code}
{
id : 1,
map:{rm: [
{mapid:m1,mapvalue:{col1:1,col2:[0,1,2,3,4,5]},rptd: [{ a:
foo},{b:boo}]},
{mapid:m2,mapvalue:{col1:0,col2:[]},rptd: [{ a:
bar},{c:1},{d:4.5}]}
]}
}
{code}
The below CTAS statement fails though the underlying query succeeds
{code}
create table rep_map as select d.map from `temp.json` d;
Query failed: Query stopped., index: -4, length: 4 (expected: range(0,
16384)) [ d76e3f74-7e2c-406f-a7fd-5efc68227e75 on qa-node190.qa.lab:31010 ]
{code}
Error from the logs :
{code}
Caused by: java.lang.IndexOutOfBoundsException: index: -4, length: 4
(expected: range(0, 16384))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at
org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:309)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:451)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.reader(RepeatedMapReaderImpl.java:60)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter$RepeatedMapParquetConverter.init(ParquetRecordWriter.java:269)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewRepeatedMapConverter(ParquetRecordWriter.java:259)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:116)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.init(ParquetRecordWriter.java:240)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewMapConverter(ParquetRecordWriter.java:230)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:114)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.initFieldWriters(EventBasedRecordWriter.java:73)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
... 19 common frames omitted
2015-02-24 19:05:56,898 [2b13346a-d929-d659-c074-354c9f5d9d4f:frag:0:0] ERROR
o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
7c8fa189-8f0a-4763-a266-fd68a11bbb66: Query stopped.
java.io.IOException: Failed to initialize FieldWriters.
at
org.apache.drill.exec.store.EventBasedRecordWriter.initFieldWriters(EventBasedRecordWriter.java:78)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.store.EventBasedRecordWriter.init(EventBasedRecordWriter.java:46)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:176)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:113)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Created] (DRILL-2797) Remove DrillRelOptUtil.createRename after Calcite rebase
Venki Korukanti created DRILL-2797:
--
Summary: Remove DrillRelOptUtil.createRename after Calcite rebase
Key: DRILL-2797
URL: https://issues.apache.org/jira/browse/DRILL-2797
Project: Apache Drill
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 1.0.0
{{DrillRelOptUtil.createRename}} was copied
{{org.eigenbase.rel.CalcRel#createRename(RelNode, List)}} as part of DRILL-2341
due to a bug Drill's fork of Calcite. The bug is fixed in v1.2 of Calcite.
Remove the copied method from Drill, once we complete Drill moves to using v1.2
version of Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2606) Casting a column from hbase to boolean after a join with parquet makes DRILL unresponsive
[
https://issues.apache.org/jira/browse/DRILL-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated DRILL-2606:
---
Attachment: DRILL-2606-2.patch
Uploading v2 patch. Added e2e test case involveing multi fragment run.
Casting a column from hbase to boolean after a join with parquet makes DRILL
unresponsive
-
Key: DRILL-2606
URL: https://issues.apache.org/jira/browse/DRILL-2606
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 0.8.0
Reporter: Rahul Challapalli
Assignee: Venki Korukanti
Priority: Critical
Fix For: 0.9.0
Attachments: DRILL-2606-1.patch, DRILL-2606-2.patch, dataload.hql,
error.log, fewtypes_null.parquet, fewtypes_null_hbase.tbl
Small scale repro without HBase:
{code}
alter session set `planner.slice_target`=1;
select cast(cast(bool_col as varchar(100)) as boolean) from
dfs.`/Users/hadoop/Downloads/fewtypes_null.parquet` order by int_col;
{code}
git.commit.id.abbrev=4d398ed
The below query never returns and causes Sqlline to hang. DRILL also becomes
unresponsive after executing this query. So I have to restart drill for
sqlline to even come up.
{code}
select cast(cast(o.types.bool_col as varchar(100)) as boolean) bool_col
from dfs.`cross-sources`.`fewtypes_null.parquet` p
inner join hbase.fewtypes_null o
on p.int_col = cast(cast(o.types.int_col as varchar(100)) as int)
{code}
This issue is similar to DRILL-2605. However in this case I did not use a
view and some this causes drill/sqlline to become unresponsive where in
DRILL-2605 we just get back an error.
I attached the data files, data loading script, and the error log. Let me
know if you need anything else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2589) Creating a view with duplicate column names should fail or give a warning to the user
[
https://issues.apache.org/jira/browse/DRILL-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496300#comment-14496300
]
Venki Korukanti commented on DRILL-2589:
First we need to address why just the query {{SELECT region_id, region_id FROM
cp.`region.json}} is allowed.
Creating a view with duplicate column names should fail or give a warning to
the user
-
Key: DRILL-2589
URL: https://issues.apache.org/jira/browse/DRILL-2589
Project: Apache Drill
Issue Type: Bug
Components: Metadata
Affects Versions: 0.8.0
Reporter: Rahul Challapalli
Assignee: Venki Korukanti
Fix For: 1.0.0
git.commit.id.abbrev=4d398ed
View DDL :
{code}
create view v1 as select pageRank, pageRank from `/dfs/parquet/rankings`;
{code}
The above view creation succeeds and drill silently renames the columns.
Postgres does not allow this behavior.
Below is the description of the view
{code}
describe v1;
+-++-+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+-++-+
| pageRank| ANY| NO |
| pageRank0 | ANY| NO |
+-++-+
{code}
In most cases the reason someone would create such a view is by mistake. So
we should either not allow it or give a warning to the user in such situations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2589) Creating a view with duplicate column names should fail or give a warning to the user
[
https://issues.apache.org/jira/browse/DRILL-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated DRILL-2589:
---
Fix Version/s: (was: 0.9.0)
1.0.0
Creating a view with duplicate column names should fail or give a warning to
the user
-
Key: DRILL-2589
URL: https://issues.apache.org/jira/browse/DRILL-2589
Project: Apache Drill
Issue Type: Bug
Components: Metadata
Affects Versions: 0.8.0
Reporter: Rahul Challapalli
Assignee: Venki Korukanti
Fix For: 1.0.0
git.commit.id.abbrev=4d398ed
View DDL :
{code}
create view v1 as select pageRank, pageRank from `/dfs/parquet/rankings`;
{code}
The above view creation succeeds and drill silently renames the columns.
Postgres does not allow this behavior.
Below is the description of the view
{code}
describe v1;
+-++-+
| COLUMN_NAME | DATA_TYPE | IS_NULLABLE |
+-++-+
| pageRank| ANY| NO |
| pageRank0 | ANY| NO |
+-++-+
{code}
In most cases the reason someone would create such a view is by mistake. So
we should either not allow it or give a warning to the user in such situations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2680) Assertions and thrown AssertionsErrors should include some message text
[ https://issues.apache.org/jira/browse/DRILL-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Barclay (Drill) updated DRILL-2680: -- Issue Type: Task (was: Bug) Assertions and thrown AssertionsErrors should include some message text --- Key: DRILL-2680 URL: https://issues.apache.org/jira/browse/DRILL-2680 Project: Apache Drill Issue Type: Task Reporter: Daniel Barclay (Drill) Assignee: Daniel Barclay (Drill) Drill seems to have a lot of assert ... statements and throw new AssertionError() cases that include no message text. Many of them should at least include a phrase or two about what's wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2680) Assertions, thrown AssertionErrors should include some message text [umbrella/tracking bug]
[ https://issues.apache.org/jira/browse/DRILL-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Barclay (Drill) updated DRILL-2680: -- Summary: Assertions, thrown AssertionErrors should include some message text [umbrella/tracking bug] (was: Assertions and thrown AssertionsErrors should include some message text) Assertions, thrown AssertionErrors should include some message text [umbrella/tracking bug] --- Key: DRILL-2680 URL: https://issues.apache.org/jira/browse/DRILL-2680 Project: Apache Drill Issue Type: Task Reporter: Daniel Barclay (Drill) Assignee: Daniel Barclay (Drill) Drill seems to have a lot of assert ... statements and throw new AssertionError() cases that include no message text. Many of them should at least include a phrase or two about what's wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2675) Implement a subset of User Exceptions to improve how errors are reported to the user
[ https://issues.apache.org/jira/browse/DRILL-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2675: Attachment: DRILL-2675.7.patch.txt Implement a subset of User Exceptions to improve how errors are reported to the user Key: DRILL-2675 URL: https://issues.apache.org/jira/browse/DRILL-2675 Project: Apache Drill Issue Type: Sub-task Components: Execution - Flow Affects Versions: 0.9.0 Reporter: Deneche A. Hakim Assignee: Jacques Nadeau Fix For: 0.9.0 Attachments: DRILL-2675.0.patch.txt, DRILL-2675.1.patch.txt, DRILL-2675.2.patch.txt, DRILL-2675.3.patch.txt, DRILL-2675.4.patch.txt, DRILL-2675.5.patch.txt, DRILL-2675.6.patch.txt, DRILL-2675.7.patch.txt Implement a set of most needed User Exceptions. Each user exception will contain: - a meaningful error message - optional context information (e.g. which file/row caused the error) - errorID and drillbit name, to help users and developers find the original error in the logs - a set of suppressed exceptions that were thrown along the original error -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2675) Implement a subset of User Exceptions to improve how errors are reported to the user
[ https://issues.apache.org/jira/browse/DRILL-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496430#comment-14496430 ] Jacques Nadeau commented on DRILL-2675: --- Merged in 99b6d0e Implement a subset of User Exceptions to improve how errors are reported to the user Key: DRILL-2675 URL: https://issues.apache.org/jira/browse/DRILL-2675 Project: Apache Drill Issue Type: Sub-task Components: Execution - Flow Affects Versions: 0.9.0 Reporter: Deneche A. Hakim Assignee: Jacques Nadeau Fix For: 0.9.0 Attachments: DRILL-2675.0.patch.txt, DRILL-2675.1.patch.txt, DRILL-2675.2.patch.txt, DRILL-2675.3.patch.txt, DRILL-2675.4.patch.txt, DRILL-2675.5.patch.txt, DRILL-2675.6.patch.txt Implement a set of most needed User Exceptions. Each user exception will contain: - a meaningful error message - optional context information (e.g. which file/row caused the error) - errorID and drillbit name, to help users and developers find the original error in the logs - a set of suppressed exceptions that were thrown along the original error -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2440) Drill should support MongoDB's extended JSON support
[
https://issues.apache.org/jira/browse/DRILL-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacques Nadeau resolved DRILL-2440.
---
Resolution: Fixed
Resolved in 314e5a2
Drill should support MongoDB's extended JSON support
Key: DRILL-2440
URL: https://issues.apache.org/jira/browse/DRILL-2440
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Affects Versions: 0.7.0
Environment: Windows embeded mode. MongoDB plugin
Reporter: Filip Grabowski
Assignee: Jacques Nadeau
Fix For: 0.9.0
I configured Apache drill and connected to Mongo and created some nice
queries.
This is a query that works fine - just a proof that the environment is
configured.
SELECT interactionDateTime FROM interactionrecord limit 5;
{ $date : 2015-03-05T00:08:10.753Z}
{ $date : 2015-03-05T00:08:10.794Z}
{ $date : 2015-03-05T00:08:10.803Z}
{ $date : 2015-03-05T00:08:35.180Z}
{ $date : 2015-03-05T00:08:35.180Z}
This simple query fails regardless of upper or lower case of function name.
Similarly fail other queries that use other data time functions. This one
seemed the simplest...
SELECT CURRENT_DATE FROM interactionrecord limit 5;
Query failed to execute. Execute Query failed.
Error type: Odbc error. Odbc operation attempted: SQLExecDirect. [S1000:1040:
on HSTMT] [MapR][Drill] (1040) Drill failed to execute the query: SELECT
CURRENT_DATE FROM interactionrecord limit 5
[30024]Query execution error. Details:[
Query stopped., Failure while trying to materialize incoming schema. Errors:
Error in expression at index -1. Error: Missing function implementation:
[extractDay(MAP-REQUIRED)]. Full expression: --UNKNOWN EXPRESSION--.. [
714416ec-e841-4443-8d70-bbb351547ddd on Window764bi-0:31010 ]
]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2754) Allocation bug in splitAndTransfer method causing some flatten queries to fail
[
https://issues.apache.org/jira/browse/DRILL-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-2754:
---
Attachment: DRILL-2754.patch
Allocation bug in splitAndTransfer method causing some flatten queries to fail
--
Key: DRILL-2754
URL: https://issues.apache.org/jira/browse/DRILL-2754
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types, Execution - Relational Operators
Reporter: Jason Altekruse
Assignee: Jason Altekruse
Priority: Critical
Fix For: 0.9.0
Attachments: DRILL-2754.patch
Data for reproduction:
{code}
{
config: [
[],
[ a string ]
]
}
{code}
{code}
select flatten(config) as flat from cp.`/store/json/null_list.json`
{code}
This was carved out of a larger use case that was failing, so in the course
of coming up with a minimal reproduction I fixed the bug. I will be posting a
patch shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2754) Allocation bug in splitAndTransfer method causing some flatten queries to fail
[
https://issues.apache.org/jira/browse/DRILL-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-2754:
---
Fix Version/s: 0.9.0
Allocation bug in splitAndTransfer method causing some flatten queries to fail
--
Key: DRILL-2754
URL: https://issues.apache.org/jira/browse/DRILL-2754
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types, Execution - Relational Operators
Reporter: Jason Altekruse
Assignee: Jason Altekruse
Priority: Critical
Fix For: 0.9.0
Attachments: DRILL-2754.patch
Data for reproduction:
{code}
{
config: [
[],
[ a string ]
]
}
{code}
{code}
select flatten(config) as flat from cp.`/store/json/null_list.json`
{code}
This was carved out of a larger use case that was failing, so in the course
of coming up with a minimal reproduction I fixed the bug. I will be posting a
patch shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-2798) Suppress log location message from sqlline
Parth Chandra created DRILL-2798: Summary: Suppress log location message from sqlline Key: DRILL-2798 URL: https://issues.apache.org/jira/browse/DRILL-2798 Project: Apache Drill Issue Type: Bug Components: Client - CLI Affects Versions: 0.8.0 Reporter: Parth Chandra Assignee: Patrick Wong Fix For: 0.9.0 sqlline is now printing a message with the location of the log file that is breaking external scripts to extract data using Drill. We need to add an option to suppress sqlline shell script messages (or remove them). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2799) Query fails if directory contains .DS_Store
[
https://issues.apache.org/jira/browse/DRILL-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496637#comment-14496637
]
Abhishek Girish commented on DRILL-2799:
Git.Commit.ID: 314e5a2 (Apr 15, 2015)
Query fails if directory contains .DS_Store
---
Key: DRILL-2799
URL: https://issues.apache.org/jira/browse/DRILL-2799
Project: Apache Drill
Issue Type: Bug
Components: Storage - Other
Reporter: Abhishek Girish
Assignee: Jacques Nadeau
On accessing a folder, Mac OS X writes .DS_Store (some metadata) into it. See
http://en.wikipedia.org/wiki/.DS_Store
When querying such a folder, Drill throws an error. Drill should ignore this
file. And there should be a way to configure the same.
{code:sql}
select * from dfs.`/data/json/factbook` limit 1;
Query failed: DATA_READ ERROR: Error reading JSON. - Invalid UTF-32 character
0x42756431(above 10) at char #1, byte #7)
Filename: /data/json/factbook/.DS_Store
Record: 1
[f73266e5-3171-4134-a0a8-671af037ddd9 on abhi6.qa.lab:31010]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Removing the file results in successfully querying the directory.
Log Snippet:
{code}
2015-04-15 11:14:04,256 [2ad15592-eb25-eb7a-5e7b-1c93e68171c6:frag:0:0] ERROR
o.a.drill.exec.ops.FragmentContext - Fragment Context received failure --
Fragment: 0:0
org.apache.drill.common.exceptions.DrillUserException: DATA_READ ERROR: Error
reading JSON. - Invalid UTF-32 character 0x42756431(above 10) at char
#1, byte #7)
Filename: /data/json/factbook/.DS_Store
Record: 1
[f73266e5-3171-4134-a0a8-671af037ddd9 on abhi6.qa.lab:31010]
at
org.apache.drill.common.exceptions.DrillUserException$Builder.build(DrillUserException.java:115)
~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader.handleAndRaise(JSONRecordReader.java:171)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:218)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:170)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2611) Value vectors report invalid value count
[ https://issues.apache.org/jira/browse/DRILL-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanifi Gunes updated DRILL-2611: Assignee: Mehant Baid (was: Hanifi Gunes) Value vectors report invalid value count Key: DRILL-2611 URL: https://issues.apache.org/jira/browse/DRILL-2611 Project: Apache Drill Issue Type: Bug Components: Execution - Data Types Reporter: Hanifi Gunes Assignee: Mehant Baid Priority: Critical Fix For: 0.9.0 We maintain an exclusive value count variable in fixed vectors however we don't update it upon calling set/Safe. Accessor reports the value count from the variable ignoring values that are already in the buffer or written via set/Safe. This causes execution failures manifested as IOOB when underlying data is sparse. We should either remove the variable and report value count directly investigating the buffer if not computationally expensive or update the variable each time we make a write to the vector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1785) Infer operator return type from function templates
[ https://issues.apache.org/jira/browse/DRILL-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanifi Gunes updated DRILL-1785: Fix Version/s: (was: 0.9.0) 1.0.0 Infer operator return type from function templates -- Key: DRILL-1785 URL: https://issues.apache.org/jira/browse/DRILL-1785 Project: Apache Drill Issue Type: Improvement Components: Functions - Drill Reporter: Hanifi Gunes Assignee: Hanifi Gunes Priority: Minor Fix For: 1.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2162) Multiple flattens on a list within a list results in violating the incoming batch size limit
[
https://issues.apache.org/jira/browse/DRILL-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496678#comment-14496678
]
Jason Altekruse commented on DRILL-2162:
The fix for this issue was actually included in the patch for DRILL-2695, so it
was fixed in 314e5a2a8f476f059153fde1b7e7da7d882db94e
Multiple flattens on a list within a list results in violating the incoming
batch size limit
Key: DRILL-2162
URL: https://issues.apache.org/jira/browse/DRILL-2162
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Fix For: 0.9.0
Attachments: data.json, drill-2162.patch
git.commit.id.abbrev=3e33880
I attached the data set with 2 records.
The below query succeeds on top of the attached data set. However when I
copied over the same data set 5 times, the same query failed
{code}
select uid, flatten(d.lst_lst[1]) lst1, flatten(d.lst_lst[0]) lst0,
flatten(d.lst_lst) lst from `data.json` d;
Query failed: RemoteRpcException: Failure while running fragment., Incoming
batch of org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch has
size 102375, which is beyond the limit of 65536 [
ef16dd95-40e2-4b66-ba30-8650ddb99812 on qa-node190.qa.lab:31010 ]
[ ef16dd95-40e2-4b66-ba30-8650ddb99812 on qa-node190.qa.lab:31010 ]
{code}
Error from the logs :
{code}
java.lang.IllegalStateException: Incoming batch of
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch has size
102375, which is beyond the limit of 65536
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:129)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext(FlattenRecordBatch.java:122)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Updated] (DRILL-2675) Implement a subset of User Exceptions to improve how errors are reported to the user
[ https://issues.apache.org/jira/browse/DRILL-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2675: Attachment: DRILL-2675.fix.patch.txt incremental patch that adds the missing changes from master Implement a subset of User Exceptions to improve how errors are reported to the user Key: DRILL-2675 URL: https://issues.apache.org/jira/browse/DRILL-2675 Project: Apache Drill Issue Type: Sub-task Components: Execution - Flow Affects Versions: 0.9.0 Reporter: Deneche A. Hakim Assignee: Jacques Nadeau Fix For: 0.9.0 Attachments: DRILL-2675.0.patch.txt, DRILL-2675.1.patch.txt, DRILL-2675.2.patch.txt, DRILL-2675.3.patch.txt, DRILL-2675.4.patch.txt, DRILL-2675.5.patch.txt, DRILL-2675.6.patch.txt, DRILL-2675.7.patch.txt, DRILL-2675.fix.patch.txt Implement a set of most needed User Exceptions. Each user exception will contain: - a meaningful error message - optional context information (e.g. which file/row caused the error) - errorID and drillbit name, to help users and developers find the original error in the logs - a set of suppressed exceptions that were thrown along the original error -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2799) Query fails if directory contains .DS_Store
Abhishek Girish created DRILL-2799:
--
Summary: Query fails if directory contains .DS_Store
Key: DRILL-2799
URL: https://issues.apache.org/jira/browse/DRILL-2799
Project: Apache Drill
Issue Type: Bug
Components: Storage - Other
Reporter: Abhishek Girish
Assignee: Jacques Nadeau
On accessing a folder, Mac OS X writes .DS_Store (some metadata) into it. See
http://en.wikipedia.org/wiki/.DS_Store
When querying such a folder, Drill throws an error. Drill should ignore this
file. And there should be a way to configure the same.
{code:sql}
select * from dfs.`/data/json/factbook` limit 1;
Query failed: DATA_READ ERROR: Error reading JSON. - Invalid UTF-32 character
0x42756431(above 10) at char #1, byte #7)
Filename: /data/json/factbook/.DS_Store
Record: 1
[f73266e5-3171-4134-a0a8-671af037ddd9 on abhi6.qa.lab:31010]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Removing the file results in successfully querying the directory.
Log Snippet:
{code}
2015-04-15 11:14:04,256 [2ad15592-eb25-eb7a-5e7b-1c93e68171c6:frag:0:0] ERROR
o.a.drill.exec.ops.FragmentContext - Fragment Context received failure --
Fragment: 0:0
org.apache.drill.common.exceptions.DrillUserException: DATA_READ ERROR: Error
reading JSON. - Invalid UTF-32 character 0x42756431(above 10) at char #1,
byte #7)
Filename: /data/json/factbook/.DS_Store
Record: 1
[f73266e5-3171-4134-a0a8-671af037ddd9 on abhi6.qa.lab:31010]
at
org.apache.drill.common.exceptions.DrillUserException$Builder.build(DrillUserException.java:115)
~[drill-common-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader.handleAndRaise(JSONRecordReader.java:171)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader.next(JSONRecordReader.java:218)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:170)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-746) Union all operator not working with tables in hive
[
https://issues.apache.org/jira/browse/DRILL-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Challapalli closed DRILL-746.
---
Verified the fix and everything looks good
Union all operator not working with tables in hive
--
Key: DRILL-746
URL: https://issues.apache.org/jira/browse/DRILL-746
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Reporter: Rahul Challapalli
Assignee: Aman Sinha
Priority: Blocker
Fix For: 0.4.0
Attachments: students.txt, unionall.ddl
Attached the DDL and a small data set that I used. The below query fails to
be parsed
select * from hive.hivestudents1 union all select * from hive.hivestudents2;
Error: exception while executing query (state=,code=0)
Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while
running query.[error_id: 2602c8e6-2654-40bd-b000-3f015e7d699f
endpoint {
address: qa-node191.qa.lab
user_port: 31010
control_port: 31011
data_port: 31012
}
error_type: 0
message: Failure while parsing sql. CannotPlanException:[ Node
[rel#125:Subset#8.PHYSICAL.SINGLETON([]).[]] could not be implemented;
planner state:
Root: rel#125:Subset#8.PHYSICAL.SINGLETON([]).[]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-2162) Multiple flattens on a list within a list results in violating the incoming batch size limit
[
https://issues.apache.org/jira/browse/DRILL-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse resolved DRILL-2162.
Resolution: Fixed
Multiple flattens on a list within a list results in violating the incoming
batch size limit
Key: DRILL-2162
URL: https://issues.apache.org/jira/browse/DRILL-2162
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Fix For: 0.9.0
Attachments: data.json, drill-2162.patch
git.commit.id.abbrev=3e33880
I attached the data set with 2 records.
The below query succeeds on top of the attached data set. However when I
copied over the same data set 5 times, the same query failed
{code}
select uid, flatten(d.lst_lst[1]) lst1, flatten(d.lst_lst[0]) lst0,
flatten(d.lst_lst) lst from `data.json` d;
Query failed: RemoteRpcException: Failure while running fragment., Incoming
batch of org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch has
size 102375, which is beyond the limit of 65536 [
ef16dd95-40e2-4b66-ba30-8650ddb99812 on qa-node190.qa.lab:31010 ]
[ ef16dd95-40e2-4b66-ba30-8650ddb99812 on qa-node190.qa.lab:31010 ]
{code}
Error from the logs :
{code}
java.lang.IllegalStateException: Incoming batch of
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch has size
102375, which is beyond the limit of 65536
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:129)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext(FlattenRecordBatch.java:122)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
[jira] [Commented] (DRILL-2680) Assertions, thrown AssertionErrors should include some message text [umbrella/tracking bug]
[
https://issues.apache.org/jira/browse/DRILL-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496606#comment-14496606
]
Daniel Barclay (Drill) commented on DRILL-2680:
---
Partial info:
{{grep -r '\bassert\b[^.][^:]*;' . --include=*.java | grep -v /target/}}
yields about 250 raw locations (raw in the sense of not filtering out cases
where a message probably isn't really needed)
At the moment I can't find cases with {{throw new AssertionError()}} that I
thought I remembered seeing.
However, in searching now, I need a number of cases of {{throw new
UnsupportedOperationException()}} that could use a message (and some of which
should be some kind of unexpected-case exception (e.g., AssertionError)), for
example:
{{noformat}}
public static MajorType overrideMinorType(final MajorType originalMajorType,
final MinorType overrideMinorType) {
switch (originalMajorType.getMode()) {
case REPEATED:
return repeated(overrideMinorType);
case OPTIONAL:
return optional(overrideMinorType);
case REQUIRED:
return required(overrideMinorType);
default:
throw new UnsupportedOperationException();
}
}
{{noformat}}
Assertions, thrown AssertionErrors should include some message text
[umbrella/tracking bug]
---
Key: DRILL-2680
URL: https://issues.apache.org/jira/browse/DRILL-2680
Project: Apache Drill
Issue Type: Task
Reporter: Daniel Barclay (Drill)
Assignee: Daniel Barclay (Drill)
Drill seems to have a lot of assert ... statements and throw new
AssertionError() cases that include no message text.
Many of them should at least include a phrase or two about what's wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2430) Improve Error Propagation (Umbrella)
[ https://issues.apache.org/jira/browse/DRILL-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2430: Fix Version/s: (was: 0.9.0) 1.0.0 Improve Error Propagation (Umbrella) Key: DRILL-2430 URL: https://issues.apache.org/jira/browse/DRILL-2430 Project: Apache Drill Issue Type: Improvement Components: Execution - Flow Affects Versions: 0.9.0 Reporter: Deneche A. Hakim Assignee: Deneche A. Hakim Fix For: 1.0.0 In many cases, when an exception is thrown, it will be reported in the logs but the client will actually get a different and more generic message that doesn't give enough information about the problem. Our goal is to provide the user with better error messages. To do so we will separate user and system exceptions: - for user exceptions the server returns enough information about the error to the client - system exceptions only contain the necessary information to help developers debug the error, e.g. error ID + drillbit IP -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2662) Exception type not being included when propagating exception message
[ https://issues.apache.org/jira/browse/DRILL-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2662: Fix Version/s: (was: 0.9.0) 1.0.0 Exception type not being included when propagating exception message Key: DRILL-2662 URL: https://issues.apache.org/jira/browse/DRILL-2662 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.8.0 Reporter: Daniel Barclay (Drill) Assignee: Deneche A. Hakim Fix For: 1.0.0 A query that tries to cast a non-numeric string (e.g., col4) to an integer fails (as expected), with the root exception being a NumberFormatException whose exception trace printout would begin with: java.lang.NumberFormatException: col4 However, one of the higher-level chained/wrapping exceptions shows up like this: Query failed: RemoteRpcException: Failure while running fragment., col4 [ 99343f97-5c70-4454-b67f-ae550b2252fb on dev-linux2:31013 ] In particular, note that the most important information, that there was a numeric syntax error, is not present in the message, even though some details (the string with the invalid syntax) is present. This usually comes from taking getMessage() of an exception rather than toString() when making a higher-level message. The toString() method normally includes the class name--and frequently the class name contains key information that is not given in the exception message. (Maybe Sun/Oracle should have always put the full information in the message part, but they didn't.) _If_ all our exceptions were just for developers, then I'd suggest always wrapping exceptions like this: throw new WrappingException( higher-level problem: + e, e ); rather than throw new WrappingException( higher-level problem: + e.getMessage(), e ); to avoid losing information. (Then the top-most exception's message string always includes all the information from the lower-level exception's message strings.) However, since that would inject class names (irrelevant to users) into message strings (shown to users), for Drill we should probably make sure that exceptions like NumberFormatException (for expected conversion errors) are always wrapped in or replaced by exceptions that are meant for users (e.g., an InvalidIntegerFormatDataException (from standard SQL exception conditions like data exception — invalid datetime format) whose message string stands on its own (independent of whether the class name appears with it)). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1891) Error message does not get propagated correctly when reading from JSON file
[
https://issues.apache.org/jira/browse/DRILL-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-1891:
Fix Version/s: (was: 0.9.0)
1.0.0
Error message does not get propagated correctly when reading from JSON file
---
Key: DRILL-1891
URL: https://issues.apache.org/jira/browse/DRILL-1891
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Affects Versions: 0.7.0
Reporter: Victoria Markman
Assignee: Deneche A. Hakim
Fix For: 1.0.0
I made a mistake in t.json file (extra colon in the last row):
{code}
{ a1: 0 , b1: a}
{ a1: 1 , b1: b}
{ a1: 2 , b1: c}
{ a1:: 3 , b1: c}
{code}
Error message below pretty much tells me everything that went wrong.
{code}
0: jdbc:drill:schema=dfs select a1 from `t.json` where a1 is not null;
Query failed: Query stopped., Unexpected character (':' (code 58)): expected
a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: org.apache.drill.exec.vector.complex.fn.JsonReader@53c10ede;
line: 3, column: 9] [ 64182782-ebba-4c6a-a963-005b8cb48339 on
atsqa4-133.qa.lab:31010 ]
{code}
However, if a result of query above is an input to any other operator, I get
this error message:
{code}
0: jdbc:drill:schema=dfs select a1 from `t.json` where a1 is not null group
by a1;
Query failed: Query failed: Failure while running fragment., You tried to do
a batch data read operation when you were in a state of STOP. You can only
do this type of operation when you are in a state of OK or OK_NEW_SCHEMA. [
955aac65-5e43-4430-baf6-ed6bb8a020d9 on atsqa4-133.qa.lab:31010 ]
[ 955aac65-5e43-4430-baf6-ed6bb8a020d9 on atsqa4-133.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Very painful for the user if query is really complex.
The same behavior if file does not exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2658) Add ilike and regex substring functions
[ https://issues.apache.org/jira/browse/DRILL-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2658: Assignee: Steven Phillips (was: Deneche A. Hakim) Add ilike and regex substring functions --- Key: DRILL-2658 URL: https://issues.apache.org/jira/browse/DRILL-2658 Project: Apache Drill Issue Type: New Feature Components: Functions - Drill Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2658.patch, DRILL-2658.patch This will not modify the parser, so postgress syntax such as: ... where c ILIKE '%ABC%' will not be currently supported. It will simply be a function: ... where ILIKE(c, '%ABC%') Same for substring: select substr(c, 'abc')... will be equivalent to postgress select substr(c from 'abc'), but 'abc' will be treated as a java regex pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1245) Drill should pinpoint to the Problem Record when it fails to parse a json file
[
https://issues.apache.org/jira/browse/DRILL-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-1245:
Fix Version/s: (was: 0.9.0)
1.0.0
Drill should pinpoint to the Problem Record when it fails to parse a json
file
Key: DRILL-1245
URL: https://issues.apache.org/jira/browse/DRILL-1245
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Reporter: Rahul Challapalli
Assignee: Deneche A. Hakim
Fix For: 1.0.0
Attachments: DRILL-1245.1.patch.txt
git.commit.id.abbrev=98b208e
Data :
{code}
{name:name1, id:1}
{name:name2, id:2}
{name:name3, id:3}
{name:name4, id:04}
{name:name5, id:5}
{code}
Query :
{code}
select * from cp.`file.json`;
Query failed: Screen received stop request sent. Invalid numeric value:
Leading zeroes not allowed
at [Source: java.io.BufferedReader@202fbdb4; line: 1, column: 24]
[c11a17bd-1a3a-4eed-a848-6d79225399d3]
Error: exception while executing query: Failure while trying to get next
result batch. (state=,code=0)
{code}
The msg should point to the exact record which is causing the problem as it
will be hard looking into the data and finding out the problem record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2658) Add ilike and regex substring functions
[ https://issues.apache.org/jira/browse/DRILL-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496638#comment-14496638 ] Deneche A. Hakim commented on DRILL-2658: - +1 Add ilike and regex substring functions --- Key: DRILL-2658 URL: https://issues.apache.org/jira/browse/DRILL-2658 Project: Apache Drill Issue Type: New Feature Components: Functions - Drill Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2658.patch, DRILL-2658.patch This will not modify the parser, so postgress syntax such as: ... where c ILIKE '%ABC%' will not be currently supported. It will simply be a function: ... where ILIKE(c, '%ABC%') Same for substring: select substr(c, 'abc')... will be equivalent to postgress select substr(c from 'abc'), but 'abc' will be treated as a java regex pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2539) NullReader should allocate an empty vector in copy* methods
[ https://issues.apache.org/jira/browse/DRILL-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanifi Gunes updated DRILL-2539: Fix Version/s: (was: 0.9.0) 1.0.0 NullReader should allocate an empty vector in copy* methods --- Key: DRILL-2539 URL: https://issues.apache.org/jira/browse/DRILL-2539 Project: Apache Drill Issue Type: Bug Components: Execution - Data Types, Execution - Flow Reporter: Hanifi Gunes Assignee: Hanifi Gunes Fix For: 1.0.0 Projecting an non-existing field from a repeated type fails with an NPE mainly because projected vector is not allocated and underlying buffer is dead. This issue proposes to allocate an empty vector in NullReader's copy* methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2022) Parquet engine falls back to new Parquet reader unnecessarily
[ https://issues.apache.org/jira/browse/DRILL-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-2022: --- Fix Version/s: (was: 0.9.0) 1.0.0 Parquet engine falls back to new Parquet reader unnecessarily --- Key: DRILL-2022 URL: https://issues.apache.org/jira/browse/DRILL-2022 Project: Apache Drill Issue Type: Improvement Components: Storage - Parquet Affects Versions: 0.8.0 Reporter: Adam Gilmore Assignee: Jason Altekruse Priority: Minor Fix For: 1.0.0 Attachments: DRILL-2022.1.patch.txt, DRILL-2022.2.patch.txt The Parquet engine falls back to the new Parquet reader whenever a Parquet file that is complex (i.e. not purely primitive types) is found. The engine should still use the faster reader when all the projected columns are primitive types and only fall back to the other reader when columns containing complex types are selected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-2680) Assertions, thrown AssertionErrors should include some message text [umbrella/tracking bug]
[
https://issues.apache.org/jira/browse/DRILL-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496606#comment-14496606
]
Daniel Barclay (Drill) edited comment on DRILL-2680 at 4/15/15 6:03 PM:
Partial info:
{{grep -r '\bassert\b[^.][^:]*;' . --include=*.java | grep -v /target/}}
yields about 250 raw locations (raw in the sense of not filtering out cases
where a message probably isn't really needed)
At the moment I can't find cases with {{throw new AssertionError()}} that I
thought I remembered seeing.
However, in searching now, I need a number of cases of {{throw new
UnsupportedOperationException()}} that could use a message (and some of which
should be some kind of unexpected-case exception (e.g., AssertionError)), for
example:
{noformat}
public static MajorType overrideMinorType(final MajorType originalMajorType,
final MinorType overrideMinorType) {
switch (originalMajorType.getMode()) {
case REPEATED:
return repeated(overrideMinorType);
case OPTIONAL:
return optional(overrideMinorType);
case REQUIRED:
return required(overrideMinorType);
default:
throw new UnsupportedOperationException();
}
}
{noformat}
was (Author: dsbos):
Partial info:
{{grep -r '\bassert\b[^.][^:]*;' . --include=*.java | grep -v /target/}}
yields about 250 raw locations (raw in the sense of not filtering out cases
where a message probably isn't really needed)
At the moment I can't find cases with {{throw new AssertionError()}} that I
thought I remembered seeing.
However, in searching now, I need a number of cases of {{throw new
UnsupportedOperationException()}} that could use a message (and some of which
should be some kind of unexpected-case exception (e.g., AssertionError)), for
example:
{{noformat}}
public static MajorType overrideMinorType(final MajorType originalMajorType,
final MinorType overrideMinorType) {
switch (originalMajorType.getMode()) {
case REPEATED:
return repeated(overrideMinorType);
case OPTIONAL:
return optional(overrideMinorType);
case REQUIRED:
return required(overrideMinorType);
default:
throw new UnsupportedOperationException();
}
}
{{noformat}}
Assertions, thrown AssertionErrors should include some message text
[umbrella/tracking bug]
---
Key: DRILL-2680
URL: https://issues.apache.org/jira/browse/DRILL-2680
Project: Apache Drill
Issue Type: Task
Reporter: Daniel Barclay (Drill)
Assignee: Daniel Barclay (Drill)
Drill seems to have a lot of assert ... statements and throw new
AssertionError() cases that include no message text.
Many of them should at least include a phrase or two about what's wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2005) Create table fails to write out a parquet file created from hive- read works fine
[
https://issues.apache.org/jira/browse/DRILL-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-2005:
Fix Version/s: (was: 0.9.0)
1.0.0
Create table fails to write out a parquet file created from hive- read works
fine
-
Key: DRILL-2005
URL: https://issues.apache.org/jira/browse/DRILL-2005
Project: Apache Drill
Issue Type: Bug
Components: Storage - Writer
Affects Versions: 0.7.0
Reporter: Ramana Inukonda Nagaraj
Assignee: Deneche A. Hakim
Fix For: 1.0.0
Attachments: hive_alltypes.parquet
Created a parquet file in hive having the following DDL
hive desc alltypesparquet;
OK
c1 int
c2 boolean
c3 double
c4 string
c5 arrayint
c6 mapint,string
c7 mapstring,string
c8 structr:string,s:int,t:double
c9 tinyint
c10 smallint
c11 float
c12 bigint
c13 arrayarraystring
c15 structr:int,s:structa:int,b:string
c16 arraystructm:mapstring,string,n:int
Time taken: 0.076 seconds, Fetched: 15 row(s)
Now tried to write the file out using drill
0: jdbc:drill: create table hive_alltypesparquet as select * from
`/user/hive/warehouse/alltypesparquet`;
Query failed: Query failed: Failure while running fragment., Attempted to
access index 2147483646 when value capacity is 4095 [
7c030695-9dee-4e15-b9a7-7c807dbd3e5f on 10.10.30.167:31010 ]
[ 7c030695-9dee-4e15-b9a7-7c807dbd3e5f on 10.10.30.167:31010 ]
Drill is able to read the file (reasonably well- see 1997,1999,2000)
0: jdbc:drill: select * from `/user/hive/warehouse/alltypesparquet`;
++++++++++++++++
| c1 | c2 | c3 | c4 | c5 | c6
| c7 | c8 | c9 |c10 |c11 |c12
|c13 |c15 |c16 |
++++++++++++++++
| null | null | null | null | {bag:[]} | {map:[]}
| {map:[]} | {} | null | null | null | null
| {bag:[]} | {s:{}} | {bag:[]} |
| -1 | false | -1.1 | [B@62369833 | {bag:[]} |
{map:[]} | {map:[]} | {} | -1 | -1 | -1.0 |
-1 | {bag:[]} | {s:{}} | {bag:[]} |
| 1 | true | 1.1| [B@6e426ea4 |
{bag:[{array_element:1},{array_element:2}]} |
{map:[{key:1,value:eA==},{key:2,value:eQ==}]} |
{map:[{key:aw==,value:dg==}]} | {r:YQ==,s:9,t:2.2} | 1
| 1 | 1.0| 1 |
{bag:[{array_element:{bag:[{array_element:YQ==},{array_element:Yg==}]}},{array_element:{bag:[{array_element:Yw==},{array_element:ZA==}]}}]}
| {r:1,s:{a:2,b:eA==}} |
{bag:[{array_element:{m:{map:[]},n:1}},{array_element:{m:{map:[{key:YQ==,value:Yg==},{key:Yw==,value:ZA==}]},n:2}}]}
|
++++++++++++++++
3 rows selected (0.101 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2725) Faster work assignment logic
[ https://issues.apache.org/jira/browse/DRILL-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-2725: --- Assignee: Steven Phillips (was: Jason Altekruse) Faster work assignment logic Key: DRILL-2725 URL: https://issues.apache.org/jira/browse/DRILL-2725 Project: Apache Drill Issue Type: Improvement Components: Storage - Other Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 0.9.0 Attachments: DRILL-2725.patch The current AssignmentCreator logic for assigning work to drillbits takes a non-negligible amount of time once the number of work units is more than a few thousand We need a new algorithm to that will cut this time down to less than a second, even for tables with more than 100K files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2219) Concurrent modification exception in TopLevelAllocator if a child allocator is added during loop in close()
[ https://issues.apache.org/jira/browse/DRILL-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2219: Fix Version/s: (was: 0.9.0) 1.0.0 Concurrent modification exception in TopLevelAllocator if a child allocator is added during loop in close() --- Key: DRILL-2219 URL: https://issues.apache.org/jira/browse/DRILL-2219 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Jason Altekruse Assignee: Chris Westin Priority: Critical Fix For: 1.0.0 Attachments: DRILL-2219-v2.patch, DRILL-2219-v3.patch, DRILL-2219-v4.patch, DRILL-2219.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2370) Missing cancellation acknowledgements leave orphaned cancelled queries around
[ https://issues.apache.org/jira/browse/DRILL-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496879#comment-14496879 ] Chris Westin commented on DRILL-2370: - [~jnadeau], it seems like your changes with the new NodeTracker handle dead nodes and take care of this. Do you agree? Missing cancellation acknowledgements leave orphaned cancelled queries around - Key: DRILL-2370 URL: https://issues.apache.org/jira/browse/DRILL-2370 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.7.0 Reporter: Chris Westin Assignee: Jacques Nadeau Fix For: 0.9.0 When a query is cancelled (say by Foreman.cancel()), Foreman moves to the CANCELLED state, which indicates that cancellation is in progress. Cancellation requests are sent to remote drill fragments. These should be replied to via QueryManager.statusUpdate() (inherited from FragmentStatusListener), which in turn updates Foreman. However, if any of the cancellation requests are not acknowledged (due to drillbit failure, RPC timeout, etc), then the being-cancelled query will stay that way indefinitely. We need to have a way to find such queries and force them to be cleaned up. This may require substituting stub listeners for some of the objects involved in case cancellation acknowledgements still arrive even later still -- these need to be safely invokable by the RPC layer even though the query they referred to is gone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2370) Missing cancellation acknowledgements leave orphaned cancelled queries around
[ https://issues.apache.org/jira/browse/DRILL-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2370: Assignee: Jacques Nadeau (was: Chris Westin) Missing cancellation acknowledgements leave orphaned cancelled queries around - Key: DRILL-2370 URL: https://issues.apache.org/jira/browse/DRILL-2370 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.7.0 Reporter: Chris Westin Assignee: Jacques Nadeau Fix For: 0.9.0 When a query is cancelled (say by Foreman.cancel()), Foreman moves to the CANCELLED state, which indicates that cancellation is in progress. Cancellation requests are sent to remote drill fragments. These should be replied to via QueryManager.statusUpdate() (inherited from FragmentStatusListener), which in turn updates Foreman. However, if any of the cancellation requests are not acknowledged (due to drillbit failure, RPC timeout, etc), then the being-cancelled query will stay that way indefinitely. We need to have a way to find such queries and force them to be cleaned up. This may require substituting stub listeners for some of the objects involved in case cancellation acknowledgements still arrive even later still -- these need to be safely invokable by the RPC layer even though the query they referred to is gone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2421) ensure all allocators for a query are descendants of a single root
[ https://issues.apache.org/jira/browse/DRILL-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2421: Fix Version/s: (was: 0.9.0) 1.0.0 ensure all allocators for a query are descendants of a single root -- Key: DRILL-2421 URL: https://issues.apache.org/jira/browse/DRILL-2421 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Chris Westin Assignee: Chris Westin Fix For: 1.0.0 In order to help improve usage tracking, allocations for a single query should all roll up to a single root. This requires that the Foreman create that root, and label it, and then pass that along to anyone else that needs to create additional sub-allocators. The patch for DRILL-2406 introduces the creation of a new allocator in QueryContext, but this is currently a child of the Drillbit's TopLevelAllocator, violating the principle above. This is a reminder to fix that after the dependencies above are available. As well as the known case in QueryContext, check to make sure other locations aren't creating new children from the DrillbitContext, but are instead using the allocator from the FragmentContext instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2643) Exception during xsort.ExternalSortBatch.cleanup (possible memory leak ?)
[
https://issues.apache.org/jira/browse/DRILL-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2643:
Fix Version/s: (was: 0.9.0)
1.0.0
Exception during xsort.ExternalSortBatch.cleanup (possible memory leak ?)
-
Key: DRILL-2643
URL: https://issues.apache.org/jira/browse/DRILL-2643
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 0.8.0
Reporter: Victoria Markman
Assignee: Chris Westin
Fix For: 1.0.0
Attachments: t1.parquet, t2.parquet
In this case j1,j2 are views created on top of parquet files, BOTH views
have order by on multiple columns in different order with nulls first/last.
Also, table in in view j1, consists of 99 parquet files. See attached
views.txt file on how to create views (make sure to create views in a
different workspace, views have the same names as tables)
{code}
select DISTINCT
COALESCE(j1.c_varchar || j2.c_varchar || 'EMPTY') as
concatentated_string
from
j1 INNER JOIN j2 ON
(j1.d18 = j2.d18)
;
{code}
The same can be reproduced with parquet files and subqueries:
(pay attention parquet files are named the same as views: j1, j2)
{code}
select DISTINCT
COALESCE(sq1.c_varchar || sq2.c_varchar || 'EMPTY') as
concatentated_string
from
(select c_varchar, c_integer from j1 order by j1.c_varchar desc nulls
first ) as sq1(c_varchar, c_integer)
INNER JOIN
(select c_varchar, c_integer from j2 order by j2.c_varchar nulls
last) as sq2(c_varchar, c_integer)
ON (sq1.c_integer = sq2.c_integer)
{code}
You do need to have sort in order to reproduce the problem.
This query works:
{code}
select DISTINCT
COALESCE(j1.c_varchar || j2.c_varchar || 'EMPTY') as
concatentated_string
from j1,j2
where j1.c_integer = j2.c_integer;
{code}
{code}
2015-04-01 00:43:42,455 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO
o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps:
Executed 99 out of 99 using 16 threads. Time: 20ms total, 2.877318ms avg, 3ms
max.
2015-04-01 00:43:42,458 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
atsqa4-136.qa.lab. Skipping affinity to that host.
2015-04-01 00:43:42,458 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO
o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps:
Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.562620ms avg, 1ms max.
2015-04-01 00:43:42,485 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING --
RUNNING
2015-04-01 00:43:45,613 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] WARN
o.a.d.e.p.i.xsort.ExternalSortBatch - Starting to merge. 32 batch groups.
Current allocated memory: 16642330
2015-04-01 00:43:45,676 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [16384] - [32768]
2015-04-01 00:43:45,676 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector
``c_varchar`(VARCHAR:OPTIONAL)_bits`(UINT1:REQUIRED). [4096] - [8192]
2015-04-01 00:43:45,679 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
2015-04-01 00:43:45,680 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector
``c_varchar`(VARCHAR:OPTIONAL)_bits`(UINT1:REQUIRED). [8192] - [16384]
2015-04-01 00:43:45,709 [2ae4c0c0-c408-3e66-4fb3-e7bf80a42bad:frag:0:0] WARN
o.a.d.exec.memory.AtomicRemainder - Tried to close remainder, but it has
already been closed
java.lang.Exception: null
at
org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:196)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at org.apache.drill.exec.memory.Accountor.close(Accountor.java:386)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:298)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.cleanup(ExternalSortBatch.java:162)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.cleanup(IteratorValidatorBatchIterator.java:148)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Updated] (DRILL-2281) Drill never returns when we use aggregate functions after a join with an order by
[
https://issues.apache.org/jira/browse/DRILL-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2281:
Fix Version/s: (was: 0.9.0)
1.0.0
Drill never returns when we use aggregate functions after a join with an
order by
-
Key: DRILL-2281
URL: https://issues.apache.org/jira/browse/DRILL-2281
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Chris Westin
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=6676f2d
The below query never returns : (Order by seems to be the culprit)
{code}
create view v1 as select uid, flatten(events) event from `data.json`;
create view v2 as select uid, flatten(transactions) transaction from
`data.json`;
select v1.uid, MAX(v2.transaction.amount), MIN(v1.event.event_time) from v1
inner join v2 on v1.uid = v2.uid where v2.transaction.trans_time 0 group by
v1.uid order by v1.uid;
{code}
There seems to be constant activity in the drillbit.log file. The below
message is continuously displayed in the log file
{code}
2015-02-20 23:35:04,450 [2b183b65-4551-bb9a-35ca-b71b9eedc4d6:frag:1:2] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
2015-02-20 23:35:04,451 [2b183b65-4551-bb9a-35ca-b71b9eedc4d6:frag:1:2] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
2015-02-20 23:35:04,451 [2b183b65-4551-bb9a-35ca-b71b9eedc4d6:frag:1:2] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
{code}
Drill returns correct data when we remove one of the agg functions or use
multiple aggs from the same side of the join. The below queries work :
{code}
select v1.uid, MAX(v2.transaction.amount) from v1 inner join v2 on v1.uid =
v2.uid where v2.transaction.trans_time 0 group by v1.uid order by v1.uid;
select v1.uid, MAX(v2.transaction.amount), MAX(v2.transaction.amount) from
v1 inner join v2 on v1.uid = v2.uid where v2.transaction.trans_time 0 group
by v1.uid order by v1.uid;
{code}
Attached the dataset which contains 2 records. I copied over the same 2
records 5 times and ran the queries on the data set. Let me know if you
need anything else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2418) Memory leak during execution if comparison function is not found
[
https://issues.apache.org/jira/browse/DRILL-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2418:
Fix Version/s: (was: 0.9.0)
1.0.0
Memory leak during execution if comparison function is not found
Key: DRILL-2418
URL: https://issues.apache.org/jira/browse/DRILL-2418
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 0.8.0
Reporter: Victoria Markman
Assignee: Chris Westin
Fix For: 1.0.0
While testing implicit cast during join, I ran into an issue where if you run
a query that throws an exception during execution, eventually, if you run
enough of those, drill will run out of memory.
Here is a query example:
{code}
select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.c_float = b.c_time
failed: RemoteRpcException: Failure while running fragment., Failure finding
function that runtime code generation expected. Signature:
compare_to_nulls_high( TIME:OPTIONAL, FLOAT4:OPTIONAL ) returns INT:REQUIRED
[ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
[ 633c8ce3-1ed2-4a0a-8248-1e3d5b4f7c0a on atsqa4-133.qa.lab:31010 ]
Test_Failed: 2015/03/10 18:34:15.0015 - Failed to execute.
{code}
If you set planner.slice_target to 1, you hit out of memory after about ~40
or so of such failures on my cluster.
{code}
select count(*) from cast_tbl_1 a, cast_tbl_2 b where a.d38 = b.c_double
Query failed: OutOfMemoryException: You attempted to create a new child
allocator with initial reservation 300 but only 916199 bytes of memory
were available.
{code}
From the drillbit.log
{code}
2015-03-10 18:34:34,588 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out
of 1 using 1 threads. Time: 1ms total, 1.190007ms avg, 1ms max.
2015-03-10 18:34:34,591 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.d.e.store.parquet.FooterGatherer - Fetch Parquet Footers: Executed 1 out
of 1 using 1 threads. Time: 0ms total, 0.953679ms avg, 0ms max.
2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
atsqa4-136.qa.lab. Skipping affinity to that host.
2015-03-10 18:34:34,627 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps:
Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.609586ms avg, 1ms max.
2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on host
atsqa4-136.qa.lab. Skipping affinity to that host.
2015-03-10 18:34:34,629 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.d.e.s.parquet.ParquetGroupScan - Load Parquet RowGroup block maps:
Executed 1 out of 1 using 1 threads. Time: 1ms total, 1.270340ms avg, 1ms max.
2015-03-10 18:34:34,684 [2b00c6c5-5525-ae65-25f8-24ea2d88ba2f:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING --
FAILED
org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
during fragment initialization: Failure while getting memory allocator for
fragment.
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:195)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:303)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.drill.common.exceptions.ExecutionSetupException:
Failure while getting memory allocator for fragment.
at
org.apache.drill.exec.ops.FragmentContext.init(FragmentContext.java:119)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.setupRootFragment(Foreman.java:535)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:307)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:511)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:186)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
[jira] [Updated] (DRILL-2750) Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state
[ https://issues.apache.org/jira/browse/DRILL-2750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2750: Fix Version/s: (was: 0.9.0) 1.0.0 Running 1 or more queries against Drillbits having insufficient DirectMem renders the Drillbits in an unusable state Key: DRILL-2750 URL: https://issues.apache.org/jira/browse/DRILL-2750 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.9.0 Environment: RHEL 6.4 Reporter: Kunal Khatua Assignee: Chris Westin Priority: Critical Fix For: 1.0.0 When running queries against a Drill cluster with limited DirectMem; if one or more queries fail due to insufficient memory, then even queries that should easily run within the allocated memory fail. The initial failure when queries with large memory requirements fail: 2015-04-10 09:57:55 [pip0] ERROR PipSQuawkling fetchRows - [ 1 / 16_par1000 ] Failure while executing query. java.sql.SQLException: Failure while executing query. at org.apache.drill.jdbc.DrillCursor.next(DrillCursor.java:144) at net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187) at org.apache.drill.jdbc.DrillResultSet.next(DrillResultSet.java:85) at PipSQuawkling.fetchRows(PipSQuawkling.java:319) at PipSQuawkling.executeTest(PipSQuawkling.java:154) at PipSQuawkling.run(PipSQuawkling.java:76) Caused by: org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment.[ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ] [ e8c657a7-93a9-415a-8641-a4fbd4836a65 on ucs-node5.perf.lab:31010 ] at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:111) at org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:100) at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:52) at org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:34) at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:57) at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194) at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at java.lang.Thread.run(Thread.java:744) After that, subsequent queries that should run, fail with the following: 2015-04-10 09:59:29 [pip0] ERROR PipSQuawkling executeQuery - [ 2 / rerun_06_par1000 ] exception while executing query: Failure while executing query. java.sql.SQLException:
[jira] [Updated] (DRILL-2699) Collect all cleanup errors before reporting a failure to the client
[ https://issues.apache.org/jira/browse/DRILL-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2699: Fix Version/s: (was: 0.9.0) 1.0.0 Collect all cleanup errors before reporting a failure to the client --- Key: DRILL-2699 URL: https://issues.apache.org/jira/browse/DRILL-2699 Project: Apache Drill Issue Type: Improvement Components: Execution - Flow Affects Versions: 0.8.0 Reporter: Deneche A. Hakim Assignee: Chris Westin Fix For: 1.0.0 If a query fails, the fragments and foreman should make sure to collect all failures and report them back to the client. Some known places where this isn't respected: - If a fragment fails, it will report the failure to the foreman before cleaning up. Any failure that happens in the cleanup process will be dropped by the foreman. - If multiple fragments fail, the Foreman will only report to the user the first failure it received and close immediately. All other failures will be dropped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2140) RPC Error querying JSON with empty nested maps
[ https://issues.apache.org/jira/browse/DRILL-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2140: Fix Version/s: (was: 0.9.0) 1.0.0 RPC Error querying JSON with empty nested maps -- Key: DRILL-2140 URL: https://issues.apache.org/jira/browse/DRILL-2140 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.7.0 Environment: Centos 4 node MapR cluster Reporter: Andries Engelbrecht Assignee: Chris Westin Fix For: 1.0.0 Attachments: drillbit.log When querying large number of documents in multiple directories with multiple JSON files in each, and some documents have no top level map that is used for a predicate, Drill produces a RPC error in the log. Query select t.retweeted_status.`user`.name as name, count(t.retweeted_status.favorited) as rt_count from `./nfl` t where t.retweeted_status.`user`.name is not null group by t.retweeted_status.`user`.name order by count(t.retweeted_status.favorited) desc limit 10; Screen Error Query failed: Query failed: Failure while running fragment., index: 0, length: 1 (expected: range(0, 0)) [ b96e3bfa-74c9-4b78-886b-9a2c3fc4ea9b on se-node13.se.lab:31010 ] [ b96e3bfa-74c9-4b78-886b-9a2c3fc4ea9b on se-node13.se.lab:31010 ] Drillbit log attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2166) left join with complex type throw ClassTransformationException
[
https://issues.apache.org/jira/browse/DRILL-2166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2166:
Fix Version/s: (was: 0.9.0)
1.0.0
left join with complex type throw ClassTransformationException
--
Key: DRILL-2166
URL: https://issues.apache.org/jira/browse/DRILL-2166
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 0.8.0
Reporter: Chun Chang
Assignee: Chris Westin
Fix For: 1.0.0
#Thu Jan 29 18:00:57 EST 2015
git.commit.id.abbrev=09f7fb2
Dataset can be downloaded from
https://s3.amazonaws.com/apache-drill/files/complex.json.gz
The following query caused the exception:
{code}
0: jdbc:drill:schema=dfs.drillTestDirComplexJ select a.id, a.soa, b.sfa[0],
b.soa[1] from `complex.json` a left outer join `complex.json` b on
a.sia[0]=b.sia[0] order by a.id limit 20;
Query failed: RemoteRpcException: Failure while running fragment., Line 35,
Column 32: No applicable constructor/method found for actual parameters int,
int, org.apache.drill.exec.vector.complex.MapVector; candidates are: public
void org.apache.drill.exec.vector.NullableTinyIntVector.copyFromSafe(int,
int, org.apache.drill.exec.vector.NullableTinyIntVector), public void
org.apache.drill.exec.vector.NullableTinyIntVector.copyFromSafe(int, int,
org.apache.drill.exec.vector.TinyIntVector) [
fbf47be8-b5fe-4d56-9488-15d45d4224e4 on qa-node117.qa.lab:31010 ]
[ fbf47be8-b5fe-4d56-9488-15d45d4224e4 on qa-node117.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
stack from drill bit.log
{code}
2015-02-04 13:37:22,117 [2b2d6eee-105b-5544-9111-83a3a356285d:frag:2:6] WARN
o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing
fragment
org.apache.drill.common.exceptions.DrillRuntimeException:
org.apache.drill.exec.exception.SchemaChangeException:
org.apache.drill.exec.exception.ClassTransformationException:
java.util.concurrent.ExecutionException:
org.apache.drill.exec.exception.ClassTransformationException: Failure
generating transformation classes for value:
package org.apache.drill.exec.test.generated;
import org.apache.drill.exec.exception.SchemaChangeException;
import org.apache.drill.exec.ops.FragmentContext;
import org.apache.drill.exec.record.RecordBatch;
import org.apache.drill.exec.record.VectorContainer;
import org.apache.drill.exec.vector.NullableBigIntVector;
import org.apache.drill.exec.vector.NullableFloat8Vector;
import org.apache.drill.exec.vector.NullableTinyIntVector;
import org.apache.drill.exec.vector.complex.MapVector;
import org.apache.drill.exec.vector.complex.RepeatedMapVector;
{code}
from forman drill bit.log
{code}
2015-02-04 13:37:22,189 [BitServer-5] ERROR
o.a.d.exec.rpc.RpcExceptionHandler - Exception in pipeline. Closing channel
between local /10.10.100.117:31012 and remote /10.10.100.120:56250
io.netty.handler.codec.DecoderException: java.lang.NullPointerException
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
[netty-codec-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[netty-codec-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:161)
[netty-codec-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
[netty-transport-4.0.24.Final.jar:4.0.24.Final]
at
[jira] [Updated] (DRILL-2074) Queries fail with OutOfMemory Exception when Hash Join Agg are turned off
[
https://issues.apache.org/jira/browse/DRILL-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2074:
Fix Version/s: (was: 0.9.0)
1.0.0
Queries fail with OutOfMemory Exception when Hash Join Agg are turned off
---
Key: DRILL-2074
URL: https://issues.apache.org/jira/browse/DRILL-2074
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Abhishek Girish
Assignee: Chris Westin
Fix For: 1.0.0
Attachments: 05_par1000.q, 05_par1000_d6e54ab.logical.plan,
05_par1000_d6e54ab.verbose.plan, drill-env.sh
Query attached.
Hash Join and Hash Agg were turned off. And the following property was added
to drill-override.conf:
sort: {
purge.threshold : 100,
external: {
batch.size : 4000,
spill: {
batch.size : 4000,
group.size : 100,
threshold : 200,
directories : [ /drill_spill ],
fs : maprfs:///
}
}
}
Query failed with the below error message:
Query failed: RemoteRpcException: Failure while running fragment., Unable to
allocate sv2 buffer after repeated attempts [
faf3044a-e14a-427b-b66d-7bcd7522ead5 on drone-42:31010 ]
[ faf3044a-e14a-427b-b66d-7bcd7522ead5 on drone-42:31010 ]
Log Snippets:
2015-01-26 20:07:33,239 atsqa8c42.qa.lab
[2b396307-2c1e-3486-90bc-fbaf09fbeb3e:frag:15:51] ERROR
o.a.d.e.w.f.AbstractStatusReporter - Error
faf3044a-e14a-427b-b66d-7bcd7522ead5: Failure while running fragment.
java.lang.RuntimeException:
org.apache.drill.exec.memory.OutOfMemoryException: Unable to allocate sv2
buffer after repeated attempts
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:309)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.JoinStatus.nextRight(JoinStatus.java:80)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:95)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:147)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Updated] (DRILL-2278) Join on a large data set results in an NPE
[
https://issues.apache.org/jira/browse/DRILL-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2278:
Fix Version/s: (was: 0.9.0)
1.0.0
Join on a large data set results in an NPE
--
Key: DRILL-2278
URL: https://issues.apache.org/jira/browse/DRILL-2278
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Chris Westin
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=6676f2d
I attached the data set which only contains 2 records. The below query works
fine on this data. However if we just copy over the same 2 records 5
times, the same query fails with an NPE.
{code}
select * from `data.json` t1 inner join `data.json` t2 on t1.uid=t2.uid;
Query failed: RemoteRpcException: Failure while running fragment.[
1cec8b9a-5c04-44d8-b1a4-bcf3a0fadcbc on qa-node190.qa.lab:31010 ]
[ 1cec8b9a-5c04-44d8-b1a4-bcf3a0fadcbc on qa-node190.qa.lab:31010 ]
{code}
Projecting a specific column works as expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2050) remote rpc exception
[
https://issues.apache.org/jira/browse/DRILL-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2050:
Fix Version/s: (was: 0.9.0)
1.0.0
remote rpc exception
Key: DRILL-2050
URL: https://issues.apache.org/jira/browse/DRILL-2050
Project: Apache Drill
Issue Type: Bug
Components: Execution - RPC
Affects Versions: 0.8.0
Reporter: Chun Chang
Assignee: Chris Westin
Fix For: 1.0.0
git.branch=8d1e1affe86a5adca3bc17eeaf7520f0d379a393
git.commit.time=20.01.2015 @ 23\:02\:03 PST
The following tpcds-implala-sf1 automation query caused RemoteRpcException.
This query works most of the time. But fails on average once in every four or
five tries. Test data can be downloaded from
http://apache-drill.s3.amazonaws.com/files/tpcds-sf1-parquet.tgz
{code}
0: jdbc:drill:schema=dfs.drillTestDir select
. . . . . . . . . . . . . . . . . . . *
. . . . . . . . . . . . . . . . . . . from
. . . . . . . . . . . . . . . . . . . (select
. . . . . . . . . . . . . . . . . . . i.i_manufact_id as imid,
. . . . . . . . . . . . . . . . . . . sum(ss.ss_sales_price) sum_sales
. . . . . . . . . . . . . . . . . . . -- avg(sum(ss.ss_sales_price))
over (partition by i.i_manufact_id) avg_quarterly_sales
. . . . . . . . . . . . . . . . . . . from
. . . . . . . . . . . . . . . . . . . item as i,
. . . . . . . . . . . . . . . . . . . store_sales as ss,
. . . . . . . . . . . . . . . . . . . date_dim as d,
. . . . . . . . . . . . . . . . . . . store as s
. . . . . . . . . . . . . . . . . . . where
. . . . . . . . . . . . . . . . . . . ss.ss_item_sk = i.i_item_sk
. . . . . . . . . . . . . . . . . . . and ss.ss_sold_date_sk =
d.d_date_sk
. . . . . . . . . . . . . . . . . . . and ss.ss_store_sk = s.s_store_sk
. . . . . . . . . . . . . . . . . . . and d.d_month_seq in (1212, 1212 +
1, 1212 + 2, 1212 + 3, 1212 + 4, 1212 + 5, 1212 + 6, 1212 + 7, 1212 + 8, 1212
+ 9, 1212 + 10, 1212 + 11)
. . . . . . . . . . . . . . . . . . . and ((i.i_category in ('Books',
'Children', 'Electronics')
. . . . . . . . . . . . . . . . . . . and i.i_class in ('personal',
'portable', 'reference', 'self-help')
. . . . . . . . . . . . . . . . . . . and i.i_brand in
('scholaramalgamalg #14', 'scholaramalgamalg #7', 'exportiunivamalg #9',
'scholaramalgamalg #9'))
. . . . . . . . . . . . . . . . . . . or (i.i_category in ('Women',
'Music', 'Men')
. . . . . . . . . . . . . . . . . . . and i.i_class in ('accessories',
'classical', 'fragrances', 'pants')
. . . . . . . . . . . . . . . . . . . and i.i_brand in ('amalgimporto
#1', 'edu packscholar #1', 'exportiimporto #1', 'importoamalg #1')))
. . . . . . . . . . . . . . . . . . . and ss.ss_sold_date_sk between
2451911 and 2452275 -- partition key filter
. . . . . . . . . . . . . . . . . . . group by
. . . . . . . . . . . . . . . . . . . i.i_manufact_id,
. . . . . . . . . . . . . . . . . . . d.d_qoy
. . . . . . . . . . . . . . . . . . . ) tmp1
. . . . . . . . . . . . . . . . . . . -- where
. . . . . . . . . . . . . . . . . . . -- case when avg_quarterly_sales 0
then abs (sum_sales - avg_quarterly_sales) / avg_quarterly_sales else null
end 0.1
. . . . . . . . . . . . . . . . . . . order by
. . . . . . . . . . . . . . . . . . . -- avg_quarterly_sales,
. . . . . . . . . . . . . . . . . . . sum_sales,
. . . . . . . . . . . . . . . . . . . tmp1.imid
. . . . . . . . . . . . . . . . . . . limit 100;
+++
|imid| sum_sales |
+++
Query failed: RemoteRpcException: Failure while running fragment., Attempted
to close accountor with 1 buffer(s) still allocatedfor QueryId:
2b401913-4292-26d4-18b0-f105afe06121, MajorFragmentId: 2, MinorFragmentId: 0.
Total 1 allocation(s) of byte size(s): 771, at stack location:
org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.takeOwnership(TopLevelAllocator.java:197)
org.apache.drill.exec.rpc.data.DataServer.handle(DataServer.java:119)
org.apache.drill.exec.rpc.data.DataServer.handle(DataServer.java:48)
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:194)
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:173)
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
[jira] [Updated] (DRILL-2274) Unable to allocate sv2 buffer after repeated attempts : JOIN, Order by used in query
[
https://issues.apache.org/jira/browse/DRILL-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2274:
Fix Version/s: (was: 0.9.0)
1.0.0
Unable to allocate sv2 buffer after repeated attempts : JOIN, Order by used
in query
Key: DRILL-2274
URL: https://issues.apache.org/jira/browse/DRILL-2274
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Chris Westin
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=6676f2d
The below query fails :
{code}
select sub1.uid from `data.json` sub1 inner join `data.json` sub2 on sub1.uid
= sub2.uid order by sub1.uid;
{code}
Error from the logs :
{code}
2015-02-20 00:24:08,431 [2b1981b0-149e-981b-f83f-512c587321d7:frag:1:2] ERROR
o.a.d.e.w.f.AbstractStatusReporter - Error
66dba4ff-644c-4400-ab84-203256dc2600: Failure while running fragment.
java.lang.RuntimeException:
org.apache.drill.exec.memory.OutOfMemoryException: Unable to allocate sv2
buffer after repeated attempts
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:307)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:97)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:116)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:303)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.drill.exec.memory.OutOfMemoryException: Unable to
allocate sv2 buffer after repeated attempts
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:516)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:305)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
... 16 common frames omitted
{code}
On a different drillbit in the cluster, I found the below message for the
same run
{code}
2015-02-20 00:24:08,435 [BitServer-6] WARN
o.a.d.exec.rpc.control.WorkEventBus - A fragment message arrived but there
was no registered listener for that message: profile {
state: FAILED
error {
error_id: 66dba4ff-644c-4400-ab84-203256dc2600
endpoint {
address: qa-node191.qa.lab
user_port: 31010
control_port: 31011
data_port: 31012
}
[jira] [Updated] (DRILL-2550) Drillbit disconnect from ZK results in drillbit being lost until restart
[ https://issues.apache.org/jira/browse/DRILL-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2550: Fix Version/s: (was: 0.9.0) 1.0.0 Drillbit disconnect from ZK results in drillbit being lost until restart Key: DRILL-2550 URL: https://issues.apache.org/jira/browse/DRILL-2550 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.8.0 Reporter: Ramana Inukonda Nagaraj Assignee: Chris Westin Priority: Minor Fix For: 1.0.0 Not quite sure if this is an issue or even if its important- maybe someone can think of a situation where this might be a bigger issue. Steps taken to recreate: 1. Startup drillbits on multiple nodes. (They all come up and form a 8 node cluster) 2. Start executing a long running query. 3. Use TCPKILL to kill all connections between one node and zookeeper port 5181. Drill seems to behave very gracefully here - I see a nice error message saying Query failed: ForemanException: One more more nodes lost connectivity during query. Identified node was atsqa6c61.qa.lab However, once I start allowing connections back the node is not brought back as part of the cluster until a drillbit restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1976) Possible Memory Leak in drill jdbc client when dealing with wide columns (5000 chars long)
[
https://issues.apache.org/jira/browse/DRILL-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-1976:
Fix Version/s: (was: 0.9.0)
1.0.0
Possible Memory Leak in drill jdbc client when dealing with wide columns
(5000 chars long)
--
Key: DRILL-1976
URL: https://issues.apache.org/jira/browse/DRILL-1976
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Reporter: Rahul Challapalli
Assignee: Chris Westin
Fix For: 1.0.0
Attachments: wide-strings.sh
git.commit.id.abbrev=b491cdb
I am seeing an execution failure when I execute the same query multiple times
(10). The data file contains 9 columns out of which 7 are wide strings
(4000-5000 chars long)
{code}
select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select
str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on
ws.tinyint_var = sub.max_ti
{code}
Below are my memory settings :
{code}
DRILL_MAX_DIRECT_MEMORY=32G
DRILL_MAX_HEAP=4G
{code}
Error From the JDBC client
{code}
select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select
str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on
ws.tinyint_var = sub.max_ti
Exception in pipeline. Closing channel between local /10.10.100.190:38179
and remote qa-node191.qa.lab/10.10.100.191:31010
io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct
buffer memory
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:151)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658)
at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
at
io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:443)
at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:187)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:165)
at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
at
io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:600)
at
io.netty.buffer.UnsafeDirectLittleEndian.writeBytes(UnsafeDirectLittleEndian.java:25)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:144)
... 13 more
Channel closed between local /10.10.100.190:38179 and remote
qa-node191.qa.lab/10.10.100.191:31010
Channel is closed, discarding remaining 255231 byte(s) in buffer.
{code}
The logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-1542) Early fragment termination causes non running intermediate fragments to error
[ https://issues.apache.org/jira/browse/DRILL-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-1542: Fix Version/s: (was: 0.9.0) 1.0.0 Early fragment termination causes non running intermediate fragments to error - Key: DRILL-1542 URL: https://issues.apache.org/jira/browse/DRILL-1542 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Jacques Nadeau Assignee: Chris Westin Fix For: 1.0.0 Caused by: java.lang.NullPointerException: null at org.apache.drill.exec.work.fragment.FragmentExecutor.receivingFragmentFinished(FragmentExecutor.java:75) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.batch.ControlHandlerImpl.receivingFragmentFinished(ControlHandlerImpl.java:174) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.work.batch.ControlHandlerImpl.handle(ControlHandlerImpl.java:76) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.rpc.control.ControlServer.handle(ControlServer.java:60) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.rpc.control.ControlServer.handle(ControlServer.java:38) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:56) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:191) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:171) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) [netty-codec-4.0.20.Final.jar:4.0.20.Final] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1942) Improve off-heap memory usage tracking
[ https://issues.apache.org/jira/browse/DRILL-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-1942: Fix Version/s: (was: 0.9.0) 1.0.0 Improve off-heap memory usage tracking -- Key: DRILL-1942 URL: https://issues.apache.org/jira/browse/DRILL-1942 Project: Apache Drill Issue Type: Improvement Components: Execution - Relational Operators Reporter: Chris Westin Assignee: Chris Westin Fix For: 1.0.0 We're using a lot more memory than we think we should. We may be leaking it, or not releasing it as soon as we could. This is a call to come up with some improved tracking so that we can get statistics out about exactly where we're using it, and whether or not we can release it earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2745) Query returns IOB Exception when JSON data with empty arrays is input to flatten function
[
https://issues.apache.org/jira/browse/DRILL-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2745:
Fix Version/s: (was: 0.9.0)
1.0.0
Query returns IOB Exception when JSON data with empty arrays is input to
flatten function
-
Key: DRILL-2745
URL: https://issues.apache.org/jira/browse/DRILL-2745
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 0.9.0
Environment: | 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580:
Exit early from HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53
EDT
Reporter: Khurram Faraaz
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.0.0
IOB Exception is returned when JSON file that has many empty arrays and
arrays with different types of data is passed to flatten function.
Tested on 4 node cluster on CentOS
{code}
0: jdbc:drill: select flatten(outkey) from `nestedJArry.json` ;
Query failed: RemoteRpcException: Failure while running fragment., index:
176, length: 4 (expected: range(0, 176)) [
2627cf84-9dfb-4077-8531-9955ecdbdec7 on centos-02.qa.lab:31010 ]
[ 2627cf84-9dfb-4077-8531-9955ecdbdec7 on centos-02.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
0: jdbc:drill: select outkey from `nestedJArry.json`;
++
| outkey |
++
|
[[100,1000,200,99,1,0,-1,10],[a,b,c,d,e,p,o,f,m,q,d,s,v],[2012-04-01,1998-02-20,2011-08-05,1992-01-01],[10:30:29.123,12:29:21.999],[sdfklgjsdlkjfghlsidhfgopiuesrtoipuertoiurtyoiurotuiydkfjlbn,bfn;waokefpqowertoipuwergklnjdfbpdsiofgoigiuewqrqiugkjehgjksdhbvkjshdfkjsdfbnlkfbkljrghljrelkhbdlkfjbgkdfjbgkndfbnkldfgklbhjdflkghjlnkoiurty984756897345609782-3458745uiyoheirluht7895e6y],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[null],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[test
string,hello world!,just do it!,houston we have a
problem],[1,2,3,4,5,6,7,8,9,0]] |
++
1 row selected (0.088 seconds)
Stack trace from drillbit.log
2015-04-09 23:54:41,965 [2ad8eebd-adb6-6f7e-469e-4bb8ca276984:frag:0:0] WARN
o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing
fragment
java.lang.IndexOutOfBoundsException: index: 176, length: 4 (expected:
range(0, 176))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:187)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:209)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.setInt(DrillBuf.java:513)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at
org.apache.drill.exec.vector.UInt4Vector$Mutator.set(UInt4Vector.java:363)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector.splitAndTransferTo(RepeatedVarCharVector.java:173)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.vector.RepeatedVarCharVector$TransferImpl.splitAndTransfer(RepeatedVarCharVector.java:200)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.test.generated.FlattenerGen1107.flattenRecords(FlattenTemplate.java:106)
~[na:na]
at
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doWork(FlattenRecordBatch.java:156)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext(FlattenRecordBatch.java:122)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
[jira] [Updated] (DRILL-1770) Flatten on top a subquery which applies flatten over kvgen results in a ClassCastException
[
https://issues.apache.org/jira/browse/DRILL-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-1770:
Fix Version/s: (was: 0.9.0)
1.0.0
Flatten on top a subquery which applies flatten over kvgen results in a
ClassCastException
--
Key: DRILL-1770
URL: https://issues.apache.org/jira/browse/DRILL-1770
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.0.0
Attachments: DRILL_1770.patch
git.commit.id.abbrev=108d29f
Dataset :
{code}
{map:{rm: [ {rptd: [{ a: foo}]}]}}
{code}
Query :
{code}
select flatten(sub.fk.`value`) from (select flatten(kvgen(map)) fk from
`nested.json`) sub;
Query failed: Failure while running fragment.,
org.apache.drill.exec.vector.NullableIntVector cannot be cast to
org.apache.drill.exec.vector.RepeatedVector
{code}
Let me know if you need more information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2264) Incorrect data when we use aggregate functions with flatten
[
https://issues.apache.org/jira/browse/DRILL-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2264:
Fix Version/s: (was: 0.9.0)
1.0.0
Incorrect data when we use aggregate functions with flatten
---
Key: DRILL-2264
URL: https://issues.apache.org/jira/browse/DRILL-2264
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Priority: Critical
Fix For: 1.0.0
git.commit.id.abbrev=6676f2d
Data Set :
{code}
{
uid:1,
lst_lst : [[1,2],[3,4]]
}
{
uid:2,
lst_lst : [[1,2],[3,4]]
}
{code}
The below query returns incorrect results :
{code}
select uid,MAX( flatten(lst_lst[1]) + flatten(lst_lst[0])) from `temp.json`
group by uid, flatten(lst_lst[1]), flatten(lst_lst[0]);
+++
|uid | EXPR$1 |
+++
| 1 | 6 |
| 1 | 6 |
| 1 | 6 |
| 1 | 6 |
| 2 | 6 |
| 2 | 6 |
| 2 | 6 |
| 2 | 6 |
+++
{code}
However if we use a sub query, drill returns the right data
{code}
select uid, MAX(l1+l2) from (select uid,flatten(lst_lst[1]) l1,
flatten(lst_lst[0]) l2 from `temp.json`) sub group by uid, l1, l2;
+++
|uid | EXPR$1 |
+++
| 1 | 4 |
| 1 | 5 |
| 1 | 5 |
| 1 | 6 |
| 2 | 4 |
| 2 | 5 |
| 2 | 5 |
| 2 | 6 |
+++
{code}
Also using a single flatten yields proper results
{code}
select uid,MAX(flatten(lst_lst[0])) from `temp.json` group by uid,
flatten(lst_lst[0]);
+++
|uid | EXPR$1 |
+++
| 1 | 1 |
| 1 | 2 |
| 2 | 1 |
| 2 | 2 |
+++
{code}
Marked it as critical since we return in-correct data. Let me know if you
have any other questions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2161) Flatten on a list within a list on a large data set results in an IOB Exception
[
https://issues.apache.org/jira/browse/DRILL-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2161:
Fix Version/s: (was: 0.9.0)
1.0.0
Flatten on a list within a list on a large data set results in an IOB
Exception
---
Key: DRILL-2161
URL: https://issues.apache.org/jira/browse/DRILL-2161
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow, Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=3e33880
I attached the data set which contains 2 records.
Below query works fine on the attached data set
{code}
0: jdbc:drill:schema=dfs.drillTestDir select uid, flatten(d.lst_lst) lst
from `data.json` d;
+++
|uid |lst |
+++
| 1 | [1,2,3,4,5] |
| 1 | [2,3,4,5,6] |
| 2 | [1,2,3,4,5] |
| 2 | [2,3,4,5,6] |
+++
{code}
However if I copy the same data set 50, 000 times, and run the same query, it
fails with IOB. Below is the contents of the log file
{code}
java.lang.IndexOutOfBoundsException: index: 16384, length: 4 (expected:
range(0, 16384))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at
org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:309)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedListVector.populateEmpties(RepeatedListVector.java:385)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedListVector.access$300(RepeatedListVector.java:54)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedListVector$Mutator.setValueCount(RepeatedListVector.java:132)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setValueCount(ProjectRecordBatch.java:248)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:181)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext(FlattenRecordBatch.java:122)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Updated] (DRILL-2356) Wrong result when ROUND function is used in expression
[
https://issues.apache.org/jira/browse/DRILL-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mehant Baid updated DRILL-2356:
---
Attachment: DRILL-2356.patch
Wrong result when ROUND function is used in expression
--
Key: DRILL-2356
URL: https://issues.apache.org/jira/browse/DRILL-2356
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 0.8.0
Reporter: Victoria Markman
Assignee: Mehant Baid
Priority: Critical
Fix For: 0.9.0
Attachments: DRILL-2356.patch, alltypes_with_nulls
Observe overflow in the expression SUM(ROUND ...)):
{code}
0: jdbc:drill:schema=dfs select
. . . . . . . . . . . . sum(c_bigint) as sum_c_bigint,
. . . . . . . . . . . . sum(ROUND(c_bigint/12))
. . . . . . . . . . . . from
. . . . . . . . . . . . alltypes_with_nulls
. . . . . . . . . . . . group by
. . . . . . . . . . . . c_varchar,
. . . . . . . . . . . . c_integer,
. . . . . . . . . . . . c_date,
. . . . . . . . . . . . c_time,
. . . . . . . . . . . . c_boolean;
+--++
| sum_c_bigint | EXPR$1 |
+--++
| -3477884857818808320 | -2147483648 |
| 0| 0 |
| 0| 0 |
| 4465148082249531392 | 2147483647 |
| 4465148082249531392 | 2147483647 |
| -3999734748766273536 | -2147483648 |
| 0| 0 |
| -449093763428515840 | -2147483648 |
| -1825551161692782592 | -2147483648 |
| -7308685202664980480 | -2147483648 |
| -6772904422084182016 | -2147483648 |
...
...
{code}
Wrapping ROUND around SUM, produces incorrect result as well:
{code}
0: jdbc:drill:schema=dfs select
. . . . . . . . . . . . sum(c_bigint) as sum_c_bigint,
. . . . . . . . . . . . ROUND(sum(c_bigint/12))
. . . . . . . . . . . . from
. . . . . . . . . . . . alltypes_with_nulls
. . . . . . . . . . . . group by
. . . . . . . . . . . . c_varchar,
. . . . . . . . . . . . c_integer,
. . . . . . . . . . . . c_date,
. . . . . . . . . . . . c_time,
. . . . . . . . . . . . c_boolean;
+--++
| sum_c_bigint | EXPR$1 |
+--++
| -3477884857818808320 | -2147483648 |
| 0| 0 |
| 0| 0 |
| 4465148082249531392 | 2147483647 |
| 4465148082249531392 | 2147483647 |
| -3999734748766273536 | -2147483648 |
| 0| 0 |
| -449093763428515840 | -2147483648 |
| -1825551161692782592 | -2147483648 |
| -7308685202664980480 | -2147483648 |
| -6772904422084182016 | -2147483648 |
...
...
{code}
If you remove ROUND function, you get correct result:
{code}
0: jdbc:drill:schema=dfs select
. . . . . . . . . . . . sum(c_bigint) as sum_c_bigint,
. . . . . . . . . . . . sum(c_bigint/12)
. . . . . . . . . . . . from
. . . . . . . . . . . . alltypes_with_nulls
. . . . . . . . . . . . group by
. . . . . . . . . . . . c_varchar,
. . . . . . . . . . . . c_integer,
. . . . . . . . . . . . c_date,
. . . . . . . . . . . . c_time,
. . . . . . . . . . . . c_boolean;
+--++
| sum_c_bigint | EXPR$1 |
+--++
| -3477884857818808320 | -289823738151567360 |
| 0| 0 |
| 0| 0 |
| 4465148082249531392 | 372095673520794282 |
| 4465148082249531392 | 372095673520794282 |
| -3999734748766273536 | -11229063856128 |
| 0| 0 |
...
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2217) Trying to flatten an empty list should return an empty result
[
https://issues.apache.org/jira/browse/DRILL-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2217:
Fix Version/s: (was: 0.9.0)
1.0.0
Trying to flatten an empty list should return an empty result
-
Key: DRILL-2217
URL: https://issues.apache.org/jira/browse/DRILL-2217
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Fix For: 1.0.0
Attachments: error.log
git.commit.id.abbrev=3d863b5
Data Set :
{code}
{empty:[[],[[]]]}
{code}
Query :
{code}
select flatten(empty) from `data1.json`;
Query failed: RemoteRpcException: Failure while running fragment.[
1b3123d9-92bc-45d5-bef8-b5f1be9def07 on qa-node191.qa.lab:31010 ]
[ 1b3123d9-92bc-45d5-bef8-b5f1be9def07 on qa-node191.qa.lab:31010 ]
{code}
I also attached the error from the logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2167) Order by on a repeated index from the output of a flatten on large no of records results in incorrect results
[
https://issues.apache.org/jira/browse/DRILL-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2167:
Fix Version/s: (was: 0.9.0)
1.0.0
Order by on a repeated index from the output of a flatten on large no of
records results in incorrect results
-
Key: DRILL-2167
URL: https://issues.apache.org/jira/browse/DRILL-2167
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Sudheesh Katkam
Priority: Critical
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=3e33880
The below query results in 26 records. Based on the data set we should
only receive 20 records.
{code}
select s.uid from (select d.uid, flatten(d.map.rm) rms from `data.json` d) s
order by s.rms.rptd[1].d;
{code}
When I removed the order by part, drill correctly reported 20 records.
{code}
select s.uid from (select d.uid, flatten(d.map.rm) rms from `data.json` d) s;
{code}
I attached the data set with 2 records. I copied over the data set 5
times and ran the queries on top of it. Let me know if you have any other
questions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2325) conf/drill-override-example.conf is outdated
[ https://issues.apache.org/jira/browse/DRILL-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2325: Fix Version/s: (was: 0.9.0) 1.0.0 conf/drill-override-example.conf is outdated Key: DRILL-2325 URL: https://issues.apache.org/jira/browse/DRILL-2325 Project: Apache Drill Issue Type: Bug Components: Tools, Build Test Reporter: Zhiyong Liu Assignee: Sudheesh Katkam Fix For: 1.0.0 The conf/drill-override-example.conf file is outdated. Properties have been added (e.g., compile), removed (e.g., cache.hazel.subnets) or otherwise modified. The file is statically tracked in distribution/src/resources/drill-override-example.conf. Ideally there should be a way to update the file programmatically when things change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2476) Handle IterOutcome.STOP in buildSchema()
[
https://issues.apache.org/jira/browse/DRILL-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2476:
Fix Version/s: (was: 0.9.0)
1.0.0
Handle IterOutcome.STOP in buildSchema()
Key: DRILL-2476
URL: https://issues.apache.org/jira/browse/DRILL-2476
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 0.7.0
Reporter: Sudheesh Katkam
Assignee: Sudheesh Katkam
Fix For: 1.0.0
There are some {{RecordBatch}} implementations like {{HashAggBatch}} that
override the {{buildSchema()}} function. The overriding functions do not
handle {{IterOutcome.STOP}}. This causes the {{FragmentContext}} to receive
two failures in some cases (linked JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2722) Query profile data not being sent/received (and web UI not updated)
[ https://issues.apache.org/jira/browse/DRILL-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2722: Fix Version/s: (was: 0.9.0) 1.0.0 Query profile data not being sent/received (and web UI not updated) --- Key: DRILL-2722 URL: https://issues.apache.org/jira/browse/DRILL-2722 Project: Apache Drill Issue Type: Bug Components: Client - HTTP, Execution - Flow, Execution - RPC Affects Versions: 0.8.0 Reporter: Chris Westin Assignee: Sudheesh Katkam Fix For: 1.0.0 Attachments: query1_foreman.log [~amansinha100] has a test case that shows that profile information is not being received (or not being sent, I'm not sure which) for a long-running query. The query appears to stop, even though cycles are being used, and it looks like work is being done. This is becoming a problem for monitoring query progress. We need to find out why the profile information isn't being updated, and fix that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2658) Add ilike and regex substring functions
[ https://issues.apache.org/jira/browse/DRILL-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips resolved DRILL-2658. Resolution: Fixed Fixed with 033d0df Add ilike and regex substring functions --- Key: DRILL-2658 URL: https://issues.apache.org/jira/browse/DRILL-2658 Project: Apache Drill Issue Type: New Feature Components: Functions - Drill Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2658.patch, DRILL-2658.patch This will not modify the parser, so postgress syntax such as: ... where c ILIKE '%ABC%' will not be currently supported. It will simply be a function: ... where ILIKE(c, '%ABC%') Same for substring: select substr(c, 'abc')... will be equivalent to postgress select substr(c from 'abc'), but 'abc' will be treated as a java regex pattern. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-1512) Avro Record Reader
[ https://issues.apache.org/jira/browse/DRILL-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips resolved DRILL-1512. Resolution: Fixed Fixed with 55a9a59 and bf3db31 Avro Record Reader -- Key: DRILL-1512 URL: https://issues.apache.org/jira/browse/DRILL-1512 Project: Apache Drill Issue Type: New Feature Reporter: Andrew Assignee: Steven Phillips Priority: Minor Labels: avro, drill Fix For: 0.9.0 Attachments: DRILL-1512.1.patch.txt, DRILL-1512.2.patch.txt, DRILL-1512.3.patch.txt, DRILL-1512.4a.patch, DRILL-1512.4b.patch Record reader implementation for Avro data files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2363) Support user impersonation in Drill
[ https://issues.apache.org/jira/browse/DRILL-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-2363: Fix Version/s: (was: 0.9.0) 1.0.0 Support user impersonation in Drill --- Key: DRILL-2363 URL: https://issues.apache.org/jira/browse/DRILL-2363 Project: Apache Drill Issue Type: New Feature Components: Execution - Flow, Metadata Reporter: Venki Korukanti Assignee: Venki Korukanti Fix For: 1.0.0 Currently Drill has no impersonation. All queries are run as the user who is running the Drillbit process and not as the user who issued the query from JDBC/ODBC. This restricts the controlling of who can query tables/views and who can create new tables/views. This is a umbrella JIRA to track impersonation feature in Drill. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2475) Handle IterOutcome.NONE correctly in operators
[ https://issues.apache.org/jira/browse/DRILL-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-2475: --- Fix Version/s: (was: 0.9.0) 1.0.0 Handle IterOutcome.NONE correctly in operators -- Key: DRILL-2475 URL: https://issues.apache.org/jira/browse/DRILL-2475 Project: Apache Drill Issue Type: Bug Components: Execution - Relational Operators Affects Versions: 0.8.0 Reporter: Venki Korukanti Assignee: Sudheesh Katkam Fix For: 1.0.0 Currently not all operators are handling the NONE (with no OK_NEW_SCHEMA) correctly. This JIRA is to go through the operators and check if it handling the NONE correctly or not and modify accordingly. (from DRILL-2453) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2475) Handle IterOutcome.NONE correctly in operators
[ https://issues.apache.org/jira/browse/DRILL-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-2475: --- Target Version/s: 1.0.0 (was: 0.9.0) Handle IterOutcome.NONE correctly in operators -- Key: DRILL-2475 URL: https://issues.apache.org/jira/browse/DRILL-2475 Project: Apache Drill Issue Type: Bug Components: Execution - Relational Operators Affects Versions: 0.8.0 Reporter: Venki Korukanti Assignee: Sudheesh Katkam Fix For: 1.0.0 Currently not all operators are handling the NONE (with no OK_NEW_SCHEMA) correctly. This JIRA is to go through the operators and check if it handling the NONE correctly or not and modify accordingly. (from DRILL-2453) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2798) Suppress log location message from sqlline
[ https://issues.apache.org/jira/browse/DRILL-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wong updated DRILL-2798: Attachment: DRILL-2798.1.patch.txt DRILL-2798.1.patch.txt - don't print message about Drill log dir unless environment variable DRILL_LOG_DEBUG=1 Suppress log location message from sqlline -- Key: DRILL-2798 URL: https://issues.apache.org/jira/browse/DRILL-2798 Project: Apache Drill Issue Type: Bug Components: Client - CLI Affects Versions: 0.8.0 Reporter: Parth Chandra Assignee: Patrick Wong Fix For: 0.9.0 Attachments: DRILL-2798.1.patch.txt sqlline is now printing a message with the location of the log file that is breaking external scripts to extract data using Drill. We need to add an option to suppress sqlline shell script messages (or remove them). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2798) Suppress log location message from sqlline
[ https://issues.apache.org/jira/browse/DRILL-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wong updated DRILL-2798: Assignee: DrillCommitter (was: Patrick Wong) Suppress log location message from sqlline -- Key: DRILL-2798 URL: https://issues.apache.org/jira/browse/DRILL-2798 Project: Apache Drill Issue Type: Bug Components: Client - CLI Affects Versions: 0.8.0 Reporter: Parth Chandra Assignee: DrillCommitter Fix For: 0.9.0 Attachments: DRILL-2798.1.patch.txt sqlline is now printing a message with the location of the log file that is breaking external scripts to extract data using Drill. We need to add an option to suppress sqlline shell script messages (or remove them). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-811) Selecting individual columns from views created using 'select * ......' query throws an error
[ https://issues.apache.org/jira/browse/DRILL-811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli closed DRILL-811. --- Verified that the issue is fixed. Thank You. Selecting individual columns from views created using 'select * ..' query throws an error - Key: DRILL-811 URL: https://issues.apache.org/jira/browse/DRILL-811 Project: Apache Drill Issue Type: Bug Components: SQL Parser Reporter: Rahul Challapalli Assignee: DrillCommitter Fix For: 0.5.0 Attachments: rankings.parquet git.commit.id.abbrev=70fab8c Follow the below steps : create view v1 as select * from `dfs/parquet/rankings`; select pageRank from v1; The query fails with the below error : message: Failure while parsing sql. IndexOutOfBoundsException:[ Index: 1, Size: 1 ] This looks like a known issue. However I couldn't find any bug related to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-792) Joining a hive table with parquet file is returning an empty result set
[ https://issues.apache.org/jira/browse/DRILL-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli closed DRILL-792. --- Verified this. Thanks for the fix Joining a hive table with parquet file is returning an empty result set --- Key: DRILL-792 URL: https://issues.apache.org/jira/browse/DRILL-792 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Reporter: Rahul Challapalli Assignee: Rahul Challapalli Priority: Critical Fix For: 0.5.0 Attachments: 792.ddl, rankings.parquet, rankings.txt, uservisits.parquet, uservisits.txt git.commit.id.abbrev=70fab8c 1. Joining a hive table with parquet results in an empty output. Check below query select rankings.pageRank pagerank from `dfs/parquet/rankings/` rankings inner join hive.uservisits uservisits on rankings.pageURL = uservisits.destinationurl 2. Joining hive table with hive table seems to work fine select rankings.pagerank pagerank from hive.rankings rankings inner join hive.uservisits uservisits on rankings.pageurl = uservisits.destinationurl I attached the parquet and text files required along with the required hive ddl. Let me know if you need more information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-806) Information_Schema.Views : ViewDefinition is missing
[ https://issues.apache.org/jira/browse/DRILL-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Challapalli closed DRILL-806. --- Information_Schema.Views : ViewDefinition is missing - Key: DRILL-806 URL: https://issues.apache.org/jira/browse/DRILL-806 Project: Apache Drill Issue Type: Bug Components: Metadata Reporter: Rahul Challapalli Assignee: Venki Korukanti Fix For: 0.4.0 git.commit.id.abbrev=70fab8c Information_Schema.Views : The view_definition column currently displays TODO: GetViewDefinition -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2725) Faster work assignment logic
[ https://issues.apache.org/jira/browse/DRILL-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-2725: --- Fix Version/s: (was: 0.9.0) 1.0.0 Faster work assignment logic Key: DRILL-2725 URL: https://issues.apache.org/jira/browse/DRILL-2725 Project: Apache Drill Issue Type: Improvement Components: Storage - Other Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2725.patch The current AssignmentCreator logic for assigning work to drillbits takes a non-negligible amount of time once the number of work units is more than a few thousand We need a new algorithm to that will cut this time down to less than a second, even for tables with more than 100K files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2725) Faster work assignment logic
[ https://issues.apache.org/jira/browse/DRILL-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-2725: --- Attachment: DRILL-2725.2.patch Address review comments. Faster work assignment logic Key: DRILL-2725 URL: https://issues.apache.org/jira/browse/DRILL-2725 Project: Apache Drill Issue Type: Improvement Components: Storage - Other Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2725.2.patch, DRILL-2725.patch The current AssignmentCreator logic for assigning work to drillbits takes a non-negligible amount of time once the number of work units is more than a few thousand We need a new algorithm to that will cut this time down to less than a second, even for tables with more than 100K files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1752) Drill cluster returns error when querying Mongo shards on an unsharded collection
[
https://issues.apache.org/jira/browse/DRILL-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-1752:
---
Fix Version/s: (was: 0.9.0)
1.0.0
Drill cluster returns error when querying Mongo shards on an unsharded
collection
-
Key: DRILL-1752
URL: https://issues.apache.org/jira/browse/DRILL-1752
Project: Apache Drill
Issue Type: Bug
Components: Storage - MongoDB
Affects Versions: 0.6.0, 0.7.0
Environment: Drill cluster on nodes with Mongo Shards
Reporter: Andries Engelbrecht
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.0.0
Attachments: DRILL-1752.patch
Query fails on a large unsharded collection in MongoDB sharded cluster with
drillbits on each node with Mongo shards.
Error message:
0: jdbc:drill:se0:5181 select * from unshard limit 2;
Query failed: Failure while setting up query. Incoming endpoints 1 is greater
than number of chunks 0 [cb2121f7-eb3e-48cd-8530-474ca76c598d]
Error: exception while executing query: Failure while trying to get next
result batch. (state=,code=0)
0: jdbc:drill:se0:5181 explain plan for select * from unshard limit 2;
+++
|text|json|
+++
| 00-00Screen
00-01 SelectionVectorRemover
00-02Limit(fetch=[2])
00-03 Scan(groupscan=[MongoGroupScan [MongoScanSpec=MongoScanSpec
[dbName=review_syn, collectionName=unshard, filters=null],
columns=[SchemaPath [`*`)
| {
head : {
version : 1,
generator : {
type : ExplainHandler,
info :
},
type : APACHE_DRILL_PHYSICAL,
options : [ ],
queue : 0,
resultMode : EXEC
},
graph : [ {
pop : mongo-scan,
@id : 3,
mongoScanSpec : {
dbName : review_syn,
collectionName : unshard,
filters : null
},
storage : {
type : mongo,
connection : mongodb://se4.dmz:27017,
enabled : true
},
columns : [ `*` ],
cost : 625000.0
}, {
pop : limit,
@id : 2,
child : 3,
first : 0,
last : 2,
initialAllocation : 100,
maxAllocation : 100,
cost : 625000.0
}, {
pop : selection-vector-remover,
@id : 1,
child : 2,
initialAllocation : 100,
maxAllocation : 100,
cost : 625000.0
}, {
pop : screen,
@id : 0,
child : 1,
initialAllocation : 100,
maxAllocation : 100,
cost : 625000.0
} ]
} |
+++
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-1751) Drill must indicate JSON All Text Mode needs to be turned on when such queries fail
[ https://issues.apache.org/jira/browse/DRILL-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-1751: --- Fix Version/s: (was: 0.9.0) 1.0.0 Drill must indicate JSON All Text Mode needs to be turned on when such queries fail --- Key: DRILL-1751 URL: https://issues.apache.org/jira/browse/DRILL-1751 Project: Apache Drill Issue Type: Improvement Components: Storage - JSON Reporter: Abhishek Girish Assignee: Jason Altekruse Priority: Critical Labels: error_message_must_fix Fix For: 1.0.0 Attachments: drillbit.log Although JSON All Text Mode is a documented option, it may not be obvious to turn this option ON on encountering an error. Query: select * from dfs.`/data/json/lastfm/lastfm_test/A/A/A/TRAAAEA128F935A30D.json` limit 1; Query failed: Failure while running fragment.[ 4331e9a7-c5b4-4e52-bece-214ffa5d06dd on abhi7.qa.lab:31010 ] Error: exception while executing query: Failure while executing query. (state=,code=0) Resolution: I tried setting JSON Text Mode and queries began to work. alter system set `store.json.all_text_mode` = true; +++ | ok | summary | +++ | true | store.json.all_text_mode updated. | +++ 1 row selected (0.136 seconds) 0: jdbc:drill:zk=10.10.103.34:5181 select * from dfs.`/data/json/lastfm/lastfm_test/A/A/A/TRAAAEA128F935A30D.json` limit 1; ++++++ results ++++++ 1 row selected (0.169 seconds) A clear message must be included in the logs and must be displayed on SQLline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2711) Package native PAM library for Linux in Drill tar.gz
[
https://issues.apache.org/jira/browse/DRILL-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Venki Korukanti updated DRILL-2711:
---
Fix Version/s: (was: 0.9.0)
1.0.0
Package native PAM library for Linux in Drill tar.gz
-
Key: DRILL-2711
URL: https://issues.apache.org/jira/browse/DRILL-2711
Project: Apache Drill
Issue Type: Sub-task
Components: Execution - RPC
Affects Versions: 0.9.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 1.0.0
PAM authenticator is added as part of the DRILL-2674. Currently it requires
manually getting the libjpam.so and setting the JVM option
{{java.library.path}}. This JIRA is to automate those steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-1728) Better error messages on Drill JSON read error
[
https://issues.apache.org/jira/browse/DRILL-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-1728:
---
Fix Version/s: (was: 0.9.0)
1.0.0
Better error messages on Drill JSON read error
--
Key: DRILL-1728
URL: https://issues.apache.org/jira/browse/DRILL-1728
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Reporter: Tomer Shiran
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.0.0
{code}
0: jdbc:drill:zk=localhost:2181 SELECT * FROM
dfs.root.`Users/tshiran/Development/demo/data/yelp/business.json` WHERE true
and REPEATED_CONTAINS(categories, 'Australian');
+-+--+++++--++++++++---+
| business_id | full_address | hours|open| categories |city
| review_count |name| longitude | state| stars|
latitude | attributes |type| neighborhoods |
+-+--+++++--++++++++---+
Query failed: Query stopeed., You tried to start when you are using a
ValueWriter of type NullableBitWriterImpl. [
e5bafa1e-6226-443d-80fd-51e18f330899 on 172.17.3.132:31010 ]
java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-1750) Querying directories with JSON files returns incomplete results
[ https://issues.apache.org/jira/browse/DRILL-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-1750: --- Fix Version/s: (was: 0.9.0) 1.0.0 Querying directories with JSON files returns incomplete results --- Key: DRILL-1750 URL: https://issues.apache.org/jira/browse/DRILL-1750 Project: Apache Drill Issue Type: Bug Components: Storage - JSON Reporter: Abhishek Girish Assignee: Jason Altekruse Priority: Critical Fix For: 1.0.0 Attachments: 1.json, 2.json, 3.json, 4.json I happened to observe that querying (select *) a directory with json files displays only fields common to all json files. All corresponding fields are displayed while querying each of the json files individually. And in some scenarios, querying the directory crashes sqlline. The example below may help make the issue clear: select * from dfs.`/data/json/tmp/1.json`; ++++ | artist | track_id | title| ++++ | Jonathan King | TRAAAEA128F935A30D | I'll Slap Your Face (Entertainment USA Theme) | ++++ 1 row selected (1.305 seconds) select * from dfs.`/data/json/tmp/2.json`; +++++ | artist | timestamp | track_id | title| +++++ | Supersuckers | 2011-08-01 20:30:17.991134 | TRAAAQN128F9353BA0 | Double Wide | +++++ 1 row selected (0.105 seconds) select * from dfs.`/data/json/tmp/3.json`; ++++ | timestamp | track_id | title| ++++ | 2011-08-01 20:30:17.991134 | TRAAAQN128F9353BA0 | Double Wide | ++++ 1 row selected (0.083 seconds) select * from dfs.`/data/json/tmp/4.json`; +++ | track_id | title| +++ | TRAAAQN128F9353BA0 | Double Wide | +++ 1 row selected (0.076 seconds) select * from dfs.`/data/json/tmp`; +++ | track_id | title| +++ | TRAAAQN128F9353BA0 | Double Wide | | TRAAAQN128F9353BA0 | Double Wide | | TRAAAEA128F935A30D | I'll Slap Your Face (Entertainment USA Theme) | | TRAAAQN128F9353BA0 | Double Wide | +++ 4 rows selected (0.121 seconds) JVM Crash occurs at times: select * from dfs.`/data/json/tmp`; ++++ | timestamp | track_id | title| ++++ | 2011-08-01 20:30:17.991134 | TRAAAQN128F9353BA0 | Double Wide | # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7f3cb99be053, pid=13943, tid=139898808436480 # # JRE version: OpenJDK Runtime Environment (7.0_65-b17) (build 1.7.0_65-mockbuild_2014_07_16_06_06-b00) # Java VM: OpenJDK 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x932053] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try ulimit -c unlimited before starting Java again # # An error report file with more information is saved as: # /tmp/jvm-13943/hs_error.log # # If you would like to submit a bug report, please include # instructions on how to reproduce the bug and visit: # http://icedtea.classpath.org/bugzilla # Aborted -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1586) NPE when the collection being queried for does not exist in Mongo DB
[ https://issues.apache.org/jira/browse/DRILL-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-1586: --- Fix Version/s: (was: 0.9.0) 1.0.0 NPE when the collection being queried for does not exist in Mongo DB Key: DRILL-1586 URL: https://issues.apache.org/jira/browse/DRILL-1586 Project: Apache Drill Issue Type: Bug Components: Storage - MongoDB Affects Versions: 0.6.0 Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Jason Altekruse Priority: Minor Fix For: 1.0.0 NPE when the collection being queried for does not exist in Mongo DB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1556) Querying JSON-converted-Parquet file throws parquet.io.ParquetDecodingException (Intermittent)
[ https://issues.apache.org/jira/browse/DRILL-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-1556: --- Fix Version/s: (was: 0.9.0) 1.0.0 Querying JSON-converted-Parquet file throws parquet.io.ParquetDecodingException (Intermittent) -- Key: DRILL-1556 URL: https://issues.apache.org/jira/browse/DRILL-1556 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Reporter: Abhishek Girish Assignee: Jason Altekruse Priority: Critical Fix For: 1.0.0 Attachments: drillbit.log Querying JSON data works at higher values for limit: select * from `yelp_academic_dataset_review.json` limit 1125458; Querying Parquet data (converted from JSON) fails at higher values for limit: create table yelp_academic_dataset_review as select * from `yelp_academic_dataset_review.json`; [success] select * from yelp_academic_dataset_review limit 4; [data] java.lang.RuntimeException: java.sql.SQLException: Failure while trying to get next result batch. Logs indicate an error in decoding the Parquet file. Drillbit.log is attached. 2014-10-20 15:21:22,739 [bf4a3f58-781b-4c89-b718-e1ef6eab6da4:frag:1:0] ERROR o.a.drill.exec.ops.FragmentContext - Fragment Context received failure. parquet.io.ParquetDecodingException: Can't read value in column [votes, funny] INT64 at value 61063 out of 61063, 61063 out of 61063 in currentPage. repetition level: 0, definition level: 2 This is at times consistent and some other times intermittent, for varied values provided to the limit clause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1479) Hbase query using sum randomly fails to execute in Drill-0.6.0
[
https://issues.apache.org/jira/browse/DRILL-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-1479:
---
Fix Version/s: (was: 0.9.0)
1.0.0
Hbase query using sum randomly fails to execute in Drill-0.6.0
Key: DRILL-1479
URL: https://issues.apache.org/jira/browse/DRILL-1479
Project: Apache Drill
Issue Type: Bug
Components: Storage - HBase
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.0.0
Attachments: hbase_sum.error, votertab
git.commit.id.abbrev=5c220e3
The below query works fine when we from sqlline. However we run this query as
part of batch which contains other queries, then this fails frequently with
the attached error. The source data voter resides in hbase
Drill Query :
{code}
select sum(cast(threecf['contributions'] as decimal(6,2))) from voter;
{code}
Attached the error log and the data files
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-1460) JsonReader fails reading files with decimal numbers and integers in the same field
[
https://issues.apache.org/jira/browse/DRILL-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-1460:
---
Fix Version/s: (was: 0.9.0)
1.0.0
JsonReader fails reading files with decimal numbers and integers in the same
field
--
Key: DRILL-1460
URL: https://issues.apache.org/jira/browse/DRILL-1460
Project: Apache Drill
Issue Type: Bug
Components: Storage - JSON
Affects Versions: 0.6.0, 0.7.0
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Jason Altekruse
Priority: Critical
Fix For: 1.0.0
Attachments: DRILL-1460.1.patch.txt, DRILL-1460.2.patch.txt
Used the following dataset :
http://thecodebarbarian.wordpress.com/2014/03/28/plugging-usda-nutrition-data-into-mongodb
Executed the following query
{noformat}select t.nutrients from dfs.usda.`usda.json` t limit 1;{noformat}
and it failed with following exception
{noformat}
2014-09-27 17:48:39,421 [b9dfbb9b-29a9-425d-801c-2e418533525f:frag:0:0] ERROR
o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
0568d90a-d7df-4a5d-87e9-8b9f718dffa4: Screen received stop request sent.
java.lang.IllegalArgumentException: You tried to write a BigInt type when you
are using a ValueWriter of type NullableFloat8WriterImpl.
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
{noformat}
{noformat}select t.nutrients[0].units from dfs.usda.`usda.json` t limit
1;{noformat}
and it failed with following exception
{noformat}
2014-09-27 17:50:04,394 [9ee8a529-17fd-492f-9cba-2d1f5842eae1:frag:0:0] ERROR
o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
c4c6bffd-b62b-4878-af1e-58db64453307: Screen received stop request sent.
java.lang.IllegalArgumentException: You tried to write a BigInt type when you
are using a ValueWriter of type NullableFloat8WriterImpl.
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204)
~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134)
[jira] [Updated] (DRILL-1082) Problems with creating views using a path
[ https://issues.apache.org/jira/browse/DRILL-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-1082: --- Fix Version/s: (was: 0.9.0) 1.0.0 Problems with creating views using a path - Key: DRILL-1082 URL: https://issues.apache.org/jira/browse/DRILL-1082 Project: Apache Drill Issue Type: Bug Components: Metadata Reporter: Krystal Assignee: Jason Altekruse Priority: Minor Fix For: 1.0.0 git.commit.id.abbrev=33c28f6 I tried to create a view with a path relative to dfs.default schema: 0: jdbc:drill:schema=dfs create view `dfs.default`.`views/votertsv.v2` as select columns[0] voter_id, columns[1] name, columns[2] age, columns[3] registration,columns[4] contributions,columns[5] voterzone,columns[6] create_time from `dfs`.`root`.`./drill/testdata/tsv/voter.tsv`; I got the following error message: +++ | ok | summary | +++ | false | Error: Failure while accessing Zookeeper | +++ 1 row selected (0.595 seconds) However, the view is actually created: [root@qa-node56 ~]# hadoop fs -ls /drill/testdata/p1tests/views Found 1 items -rwxr-xr-x 3 mapr mapr683 2014-06-26 09:27 /drill/testdata/p1tests/views/votertsv.v2.view.drill I cannot query from the view: 0: jdbc:drill:schema=dfs select * from `dfs.default`.`views/votertsv.v2`; Failure while parsing sql. ValidationException:[ org.eigenbase.util.EigenbaseContextException: From line 1, column 15 to line 1, column 27 ] EigenbaseContextException:[ From line 1, column 15 to line 1, column 27 ] SqlValidatorException:[ Table 'dfs.default.views/votertsv.v2' not found ] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-996) Build - allow other versions of Hadoop to be specified on command-line
[ https://issues.apache.org/jira/browse/DRILL-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-996: -- Fix Version/s: (was: 0.9.0) 1.0.0 Build - allow other versions of Hadoop to be specified on command-line -- Key: DRILL-996 URL: https://issues.apache.org/jira/browse/DRILL-996 Project: Apache Drill Issue Type: Improvement Components: Tools, Build Test Reporter: Patrick Wong Assignee: Jason Altekruse Priority: Minor Fix For: 1.0.0 Attachments: DRILL-996.1.patch.txt, DRILL-996.2.patch.txt Right now, in order to change the version of Hadoop that Drill is built against, you have to edit the POM. However, for automated build systems that build Drill against a variety of possible Hadoop versions, it would be much cleaner to be able to specify it on the command-line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1159) query a particular row within a csv file caused IllegalStateException
[ https://issues.apache.org/jira/browse/DRILL-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse updated DRILL-1159: --- Fix Version/s: (was: 0.9.0) 1.0.0 query a particular row within a csv file caused IllegalStateException - Key: DRILL-1159 URL: https://issues.apache.org/jira/browse/DRILL-1159 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Chun Chang Assignee: Jason Altekruse Priority: Minor Fix For: 1.0.0 Attachments: jira1159 #Mon Jul 14 10:10:52 PDT 2014 git.commit.id.abbrev=699851b I have some data in a csv file format. And the following query caused IllegalStateException: 0: jdbc:drill:schema=dfs select * from dfs.`bugsb.csv` where columns[0]=887; Error: exception while executing query (state=,code=0) Data is sensitive so not shown here. But I will paste stack. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2208) Error message must be updated when query contains operations on a flattened column
[
https://issues.apache.org/jira/browse/DRILL-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Altekruse updated DRILL-2208:
---
Fix Version/s: (was: 0.9.0)
1.0.0
Error message must be updated when query contains operations on a flattened
column
--
Key: DRILL-2208
URL: https://issues.apache.org/jira/browse/DRILL-2208
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.8.0
Reporter: Abhishek Girish
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.0.0
Attachments: drillbit_flatten.log
Currently i observe that if there is a flatten/kvgen operation applied on a
column, no further operations can be performed on the said column unless it
is wrapped inside a nested query.
Consider a simple flatten/kvgen operation on a complex JSON file :
select flatten(kvgen(f.`people`)) as p from `factbook/world.json` f limit 1;
++
| p |
++
| {key:languages,value:{text:Mandarin Chinese 12.44%, Spanish 4.85%,
English 4.83%, Arabic 3.25%, Hindi 2.68%, Bengali 2.66%, Portuguese 2.62%,
Russian 2.12%, Japanese 1.8%, Standard German 1.33%, Javanese 1.25% (2009
est.),note_1:percents are for \first language\ speakers only; the six
UN languages - Arabic, Chinese (Mandarin), English, French, Russian, and
Spanish (Castilian) - are the mother tongue or second language of about half
of the world's population, and are the official languages in more than half
the states in the world; some 150 to 200 languages have more than a million
speakers,note_2:all told, there are an estimated 7,100 languages spoken
in the world; aproximately 80% of these languages are spoken by less than
100,000 people; about 50 languages are spoken by only 1 person; communities
that are isolated from each other in mountainous regions often develop
multiple languages; Papua New Guinea, for example, boasts about 836 separate
languages,note_3:approximately 2,300 languages are spoken in Asia, 2,150,
in Africa, 1,311 in the Pacific, 1,060 in the Americas, and 280 in Europe}} |
| {key:religions,value:{text:Christian 33.39% (of which Roman
Catholic 16.85%, Protestant 6.15%, Orthodox 3.96%, Anglican 1.26%), Muslim
22.74%, Hindu 13.8%, Buddhist 6.77%, Sikh 0.35%, Jewish 0.22%, Baha'i 0.11%,
other religions 10.95%, non-religious 9.66%, atheists 2.01% (2010 est.)}} |
| {key:population,value:{text:7,095,217,980 (July 2013
est.),top_ten_most_populous_countries_in_millions:China 1,349.59; India
1,220.80; United States 316.67; Indonesia 251.16; Brazil 201.01; Pakistan
193.24; Nigeria 174.51; Bangladesh 163.65; Russia 142.50; Japan 127.25}} |
| {key:age_structure,value:{0_14_years:26% (male 953,496,513/female
890,372,474),15_24_years:16.8% (male 614,574,389/female
579,810,490),25_54_years:40.6% (male 1,454,831,900/female
1,426,721,773),55_64_years:8.4% (male 291,435,881/female
305,185,398),65_years_and_over:8.2% (male 257,035,416/female 321,753,746)
(2013 est.)}} |
| {key:dependency_ratios,value:{total_dependency_ratio:52
%,youth_dependency_ratio:39.9 %,elderly_dependency_ratio:12.1
%,potential_support_ratio:8.3 (2013)}} |
++
*Adding a WHERE clause with conditions on this column fails:*
select flatten(kvgen(f.`people`)) as p from `factbook/world.json` f where
f.p.`key` = 'languages';
Query failed: RemoteRpcException: Failure while running fragment., languages
[ 686bcd40-c23b-448c-93d8-b98a3b092657 on abhi5.qa.lab:31010 ]
[ 686bcd40-c23b-448c-93d8-b98a3b092657 on abhi5.qa.lab:31010 ]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
Logs indicate a NumberFormat Exception in the above case.
*And query fails to parse in the below case*
select flatten(kvgen(f.`people`)).`value` as p from `factbook/world.json` f
limit 5;
Query failed: ParseException: Encountered . at line 1, column 34.
Was expecting one of:
FROM ...
, ...
AS ...
OVER ...
Error: exception while executing query: Failure while executing query.
(state=,code=0)
Rewriting using an inner query succeeds:
select g.p.`value`.`note_3` from (select flatten(kvgen(f.`people`)) as p from
`factbook/world.json` f) g where g.p.`key`='languages';
++
| EXPR$0 |
++
| approximately 2,300 languages are spoken in Asia, 2,150, in Africa, 1,311
in the Pacific, 1,060 in the Americas, and 280 in Europe |
++
*In both the failure cases the error message needs to be updated to indicate
that the operation is not supported. The current error message and logs are
not clear for an end user. *
--
This message was sent by
[jira] [Commented] (DRILL-2161) Flatten on a list within a list on a large data set results in an IOB Exception
[
https://issues.apache.org/jira/browse/DRILL-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496730#comment-14496730
]
Jason Altekruse commented on DRILL-2161:
Many of these have been fixed in various commits, will update soon with a
smaller list of the queries that are still failing.
Flatten on a list within a list on a large data set results in an IOB
Exception
---
Key: DRILL-2161
URL: https://issues.apache.org/jira/browse/DRILL-2161
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow, Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Jason Altekruse
Fix For: 0.9.0
Attachments: data.json
git.commit.id.abbrev=3e33880
I attached the data set which contains 2 records.
Below query works fine on the attached data set
{code}
0: jdbc:drill:schema=dfs.drillTestDir select uid, flatten(d.lst_lst) lst
from `data.json` d;
+++
|uid |lst |
+++
| 1 | [1,2,3,4,5] |
| 1 | [2,3,4,5,6] |
| 2 | [1,2,3,4,5] |
| 2 | [2,3,4,5,6] |
+++
{code}
However if I copy the same data set 50, 000 times, and run the same query, it
fails with IOB. Below is the contents of the log file
{code}
java.lang.IndexOutOfBoundsException: index: 16384, length: 4 (expected:
range(0, 16384))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
at
org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:309)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedListVector.populateEmpties(RepeatedListVector.java:385)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedListVector.access$300(RepeatedListVector.java:54)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.vector.complex.RepeatedListVector$Mutator.setValueCount(RepeatedListVector.java:132)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.setValueCount(ProjectRecordBatch.java:248)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:181)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:134)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext(FlattenRecordBatch.java:122)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Closed] (DRILL-1964) Missing key elements in returned array of maps
[
https://issues.apache.org/jira/browse/DRILL-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanifi Gunes closed DRILL-1964.
---
Resolution: Not A Problem
Missing key elements in returned array of maps
--
Key: DRILL-1964
URL: https://issues.apache.org/jira/browse/DRILL-1964
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types
Affects Versions: 0.8.0
Reporter: Chun Chang
Assignee: Hanifi Gunes
Priority: Minor
Fix For: 0.9.0
#Wed Jan 07 18:54:07 EST 2015
git.commit.id.abbrev=35a350f
For an array of maps, if the schema for each map is not identical, with
today's implementation, we suppose to display each map with all elements
(keys) from all maps. This is not happening. For example, I have the
following data:
{code}
{
id: 2,
oooa: {
oa: {
oab: {
oabc: [
{
rowId: 2
},
{
rowValue1: [{rv1:1, rv2:2}, {rva1:3,
rva2:4}],
rowValue2: [{rw1:1, rw2:2}, {rwa1:3,
rwa2:4}]
}
]
}
}
}
}
{code}
The following query gives:
{code}
0: jdbc:drill:schema=dfs.drillTestDirComplexJ select t.oooa.oa.oab.oabc from
`jira2file/jira1.json` t;
++
| EXPR$0 |
++
|
[{rowId:2,rowValue1:[],rowValue2:[]},{rowValue1:[{rv1:1,rv2:2},{rva1:3,rva2:4}],rowValue2:[{rw1:1,rw2:2},{rwa1:3,rwa2:4}]}]
|
++
{code}
The returned result in a nicely formatted json form:
{code}
[
{
rowId: 2,
rowValue1: [],
rowValue2: []
},
{
rowValue1: [
{
rv1: 1,
rv2: 2
},
{
rva1: 3,
rva2: 4
}
],
rowValue2: [
{
rw1: 1,
rw2: 2
},
{
rwa1: 3,
rwa2: 4
}
]
}
]
{code}
Notice the first map includes all keys from all maps. But the second map is
missing the rowId key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-1964) Missing key elements in returned array of maps
[
https://issues.apache.org/jira/browse/DRILL-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14496707#comment-14496707
]
Hanifi Gunes commented on DRILL-1964:
-
This is an intended feature of Drill. We don't differentiate null fields from
non-existence ones. Outputting null is semantically the same with outputting
nothing. I am going to close this issue.
Missing key elements in returned array of maps
--
Key: DRILL-1964
URL: https://issues.apache.org/jira/browse/DRILL-1964
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types
Affects Versions: 0.8.0
Reporter: Chun Chang
Assignee: Hanifi Gunes
Priority: Minor
Fix For: 0.9.0
#Wed Jan 07 18:54:07 EST 2015
git.commit.id.abbrev=35a350f
For an array of maps, if the schema for each map is not identical, with
today's implementation, we suppose to display each map with all elements
(keys) from all maps. This is not happening. For example, I have the
following data:
{code}
{
id: 2,
oooa: {
oa: {
oab: {
oabc: [
{
rowId: 2
},
{
rowValue1: [{rv1:1, rv2:2}, {rva1:3,
rva2:4}],
rowValue2: [{rw1:1, rw2:2}, {rwa1:3,
rwa2:4}]
}
]
}
}
}
}
{code}
The following query gives:
{code}
0: jdbc:drill:schema=dfs.drillTestDirComplexJ select t.oooa.oa.oab.oabc from
`jira2file/jira1.json` t;
++
| EXPR$0 |
++
|
[{rowId:2,rowValue1:[],rowValue2:[]},{rowValue1:[{rv1:1,rv2:2},{rva1:3,rva2:4}],rowValue2:[{rw1:1,rw2:2},{rwa1:3,rwa2:4}]}]
|
++
{code}
The returned result in a nicely formatted json form:
{code}
[
{
rowId: 2,
rowValue1: [],
rowValue2: []
},
{
rowValue1: [
{
rv1: 1,
rv2: 2
},
{
rva1: 3,
rva2: 4
}
],
rowValue2: [
{
rw1: 1,
rw2: 2
},
{
rwa1: 3,
rwa2: 4
}
]
}
]
{code}
Notice the first map includes all keys from all maps. But the second map is
missing the rowId key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-2801) ORDER BY produces extra records
Sudheesh Katkam created DRILL-2801:
--
Summary: ORDER BY produces extra records
Key: DRILL-2801
URL: https://issues.apache.org/jira/browse/DRILL-2801
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 0.8.0
Reporter: Sudheesh Katkam
Assignee: Chris Westin
Priority: Critical
Attachments: data.csv
Running in embedded mode on my mac.
{code}
$ wc -w data.csv
5 data.csv
{code}
Here's the query:
{code}
0: jdbc:drill:zk=local SELECT count(*) FROM dfs.`data.csv`;
++
| EXPR$0 |
++
| 5 |
++
1 row selected (0.223 seconds)
0: jdbc:drill:zk=local SELECT columns[0] FROM dfs.`data.csv` ORDER BY
columns[0];
++
| EXPR$0 |
++
...
| 6 |
++
50,001 rows selected (0.928 seconds)
0: jdbc:drill:zk=local SELECT tab.col, COUNT(tab.col) FROM (SELECT columns[0]
col FROM dfs.`data.csv` ORDER BY columns[0]) tab GROUP BY tab.col;
+++
| col | EXPR$1 |
+++
| 2 | 1 |
| 3 | 1 |
| 4 | 1 |
| 5 | 10001 |
| 6 | 1 |
+++
5 rows selected (0.704 seconds)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-1441) Replace sql function requires backticks to avoid parse error
[ https://issues.apache.org/jira/browse/DRILL-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra updated DRILL-1441: - Component/s: (was: Functions - Drill) SQL Parser Target Version/s: 1.1.0 Fix Version/s: (was: 0.9.0) 1.1.0 Replace sql function requires backticks to avoid parse error Key: DRILL-1441 URL: https://issues.apache.org/jira/browse/DRILL-1441 Project: Apache Drill Issue Type: Bug Components: SQL Parser Reporter: Krystal Assignee: Jinfeng Ni Priority: Minor Fix For: 1.1.0 git.commit.id.abbrev=f8d38b6 Currently the replace function requires to be enclosed within backticks (ie. `replace`); otherwise it would fail during the parsing. For example the following query fails: 0: jdbc:drill:schema=M7 select replace(substring(name,1,10), 'or', 'EA') from `dfs.default`.voter where name like 'victor%'; Query failed: Failure while parsing sql. Encountered replace at line 1, column 8. Was expecting one of: UNION ... INTERSECT ... EXCEPT ... . . . The following modified query succeeds: 0: jdbc:drill:schema=M7 select `replace`(substring(name,1,10), 'or', 'EA') from `dfs.default`.voter where name like 'victor%'; ++ | EXPR$0 | ++ | victEA tho | | victEA you | | victEA rob | | victEA van | | victEA rob | . . . The parse error was originally filed in this jira: https://issues.apache.org/jira/browse/DRILL-738. We should find a way to avoid requiring backticks as this is a standard sql function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1331) Aggregate Statistics function - regr_avgx(X, Y) regr_avgy(X, Y)
[ https://issues.apache.org/jira/browse/DRILL-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra updated DRILL-1331: - Component/s: (was: Functions - Drill) SQL Parser Target Version/s: 1.1.0 Fix Version/s: (was: 0.9.0) 1.1.0 Aggregate Statistics function - regr_avgx(X, Y) regr_avgy(X, Y) --- Key: DRILL-1331 URL: https://issues.apache.org/jira/browse/DRILL-1331 Project: Apache Drill Issue Type: Improvement Components: SQL Parser Reporter: Yash Sharma Assignee: Jinfeng Ni Priority: Minor Fix For: 1.1.0 Attachments: DRILL-1331.patch, DRILL-1331.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1331) Aggregate Statistics function - regr_avgx(X, Y) regr_avgy(X, Y)
[ https://issues.apache.org/jira/browse/DRILL-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Chandra updated DRILL-1331: - Component/s: (was: SQL Parser) Functions - Drill Aggregate Statistics function - regr_avgx(X, Y) regr_avgy(X, Y) --- Key: DRILL-1331 URL: https://issues.apache.org/jira/browse/DRILL-1331 Project: Apache Drill Issue Type: Improvement Components: Functions - Drill Reporter: Yash Sharma Assignee: Jinfeng Ni Priority: Minor Fix For: 1.1.0 Attachments: DRILL-1331.patch, DRILL-1331.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-2803) Severe skew due to null values in columns even when other columns are non-null
Aman Sinha created DRILL-2803:
-
Summary: Severe skew due to null values in columns even when other
columns are non-null
Key: DRILL-2803
URL: https://issues.apache.org/jira/browse/DRILL-2803
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 0.8.0
Reporter: Aman Sinha
Assignee: Jacques Nadeau
If you have 2 columns that are hashed (either for distribution or for hash
based operators) and one of those columns has lots of null values, it can
result in substantial skew even if the other column has non-null values.
In the following query the combined hash value of 2 columns is 0 even when 1
column is non-null. The reason is that if the starting value is null (for
cr_reason_sk all values are null in the above query), it does not matter what
seed is passed in. The hash function treats the second parameter as a seed
and not as a combiner, so it gets ignored.
{code}
select cr_call_center_sk, cr_reason_sk, hash64(cr_reason_sk,
hash64(cr_call_center_sk)) as hash_value from catalog_returns where
cr_reason_sk is null and cr_call_center_sk is not null limit 10;
+---+--++
| cr_call_center_sk | cr_reason_sk | hash_value |
+---+--++
| 1 | null | 0 |
| 1 | null | 0 |
| 4 | null | 0 |
| 1 | null | 0 |
| 4 | null | 0 |
| 2 | null | 0 |
| 2 | null | 0 |
| 2 | null | 0 |
| 2 | null | 0 |
| 2 | null | 0 |
+---+--++
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2552) ZK disconnect to foreman node results in hung query on client
[ https://issues.apache.org/jira/browse/DRILL-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau updated DRILL-2552: -- Fix Version/s: (was: 0.9.0) 1.0.0 ZK disconnect to foreman node results in hung query on client - Key: DRILL-2552 URL: https://issues.apache.org/jira/browse/DRILL-2552 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.8.0 Reporter: Ramana Inukonda Nagaraj Assignee: Jacques Nadeau Fix For: 1.0.0 Steps taken to recreate: 1. Startup drillbits on multiple nodes. (They all come up and form a 8 node cluster) 2. Start executing a long running query. 3. Use TCPKILL to kill all connections between foreman node and zookeeper port 5181. Drill seems to detect the node as gone and cancels the query but there is no communication of this back to the client which is hanging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2218) Constant folding rule exposing planning bugs and not being used in plan where the constant expression is in the select list
[
https://issues.apache.org/jira/browse/DRILL-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Sinha updated DRILL-2218:
--
Fix Version/s: (was: 0.9.0)
1.0.0
Constant folding rule exposing planning bugs and not being used in plan where
the constant expression is in the select list
---
Key: DRILL-2218
URL: https://issues.apache.org/jira/browse/DRILL-2218
Project: Apache Drill
Issue Type: Improvement
Components: Query Planning Optimization
Reporter: Jason Altekruse
Assignee: Aman Sinha
Fix For: 1.0.0
This test method and rule is not currently in the master branch, but it does
appear in the patch posted for constant expression folding during planning,
DRILL-2060. Once it is merged, the test
TestConstantFolding.testConstExprFolding_InSelect() which is currently
ignored, will be failing. The issue is that even though the constant folding
rule for project is firing, and I have traced it to see that a replacement
project with a literal is created, it is not being selected in the final
plan. This seems rather odd, as there is a comment in the last line of the
onMatch() method of the rule that says the following. This does not appear to
be having the desired effect, may need to file a bug in calcite.
{code}
// New plan is absolutely better than old plan.
call.getPlanner().setImportance(project, 0.0);
{code}
Here is the query from the test, I expect the sum to be folded in planning
with the newly enabled project constant folding rule.
{code}
select columns[0], 3+5 from cp.`test_input.csv`
{code}
There also some planning bugs that are exposed when this rule is enabled,
even if the ReduceExpressionsRule.PROJECT_INSTANCE has no impact on the plan
itself.
It is causing a planning bug for the TestAggregateFunctions.testDrill2092 -as
well as TestProjectPushDown.testProjectPastJoinPastFilterPastJoinPushDown()-.
The rule's OnMatch is being called, but not modifying the plan. It seems like
its presence in the optimizer is making another rule fire that is creating a
bad plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
