[jira] [Resolved] (DRILL-2545) Killing a JDBC client program does not kill the query on drillbits
[ https://issues.apache.org/jira/browse/DRILL-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau resolved DRILL-2545. --- Resolution: Fixed Resolved in 960f876 Killing a JDBC client program does not kill the query on drillbits -- Key: DRILL-2545 URL: https://issues.apache.org/jira/browse/DRILL-2545 Project: Apache Drill Issue Type: Bug Components: Client - JDBC Affects Versions: 0.8.0 Reporter: Ramana Inukonda Nagaraj Assignee: Jacques Nadeau Priority: Critical Fix For: 1.0.0 Steps to repro: 1. Start running a query through sqlline or any JDBC app on a large dataset (tpch sf100) 2. Kill the JDBC app or sqlline. 3. The query still hangs around in state RUNNING through the profiles page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3006) CTAS with interval data type creates invalid parquet file
Mehant Baid created DRILL-3006: -- Summary: CTAS with interval data type creates invalid parquet file Key: DRILL-3006 URL: https://issues.apache.org/jira/browse/DRILL-3006 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Reporter: Mehant Baid Assignee: Steven Phillips Used the below CTAS statement: create table t6 as select interval '10' day interval_day_col from cp.`employee.json` limit 1; When I query the table 't6' the following exception is encountered: Caused by: java.io.IOException: Failure while trying to get footer for file file:/tmp/t6/0_0_0.parquet at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.convertToIOException(FooterGatherer.java:120) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.getValue(TimedRunnable.java:67) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:136) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer.getFooters(FooterGatherer.java:95) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooterHelper(ParquetGroupScan.java:229) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan.access$000(ParquetGroupScan.java:79) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan$1.run(ParquetGroupScan.java:206) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan$1.run(ParquetGroupScan.java:204) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) ~[hadoop-common-2.4.1.jar:na] at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooter(ParquetGroupScan.java:204) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] ... 21 common frames omitted Caused by: java.lang.IllegalArgumentException: Invalid FIXED_LEN_BYTE_ARRAY length: 0 at parquet.Preconditions.checkArgument(Preconditions.java:50) ~[parquet-common-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$PrimitiveBuilder.build(Types.java:320) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$PrimitiveBuilder.build(Types.java:250) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$Builder.named(Types.java:228) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:640) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:601) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:529) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:480) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at org.apache.drill.exec.store.parquet.FooterGatherer.readFooter(FooterGatherer.java:161) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:115) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:102) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:47) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:107) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] ... 30 common frames omitted When I run parquet tools (parquet-schema or parquet-meta) on the parquet file I get a similar error: Invalid FIXED_LEN_BYTE_ARRAY
[jira] [Commented] (DRILL-3006) CTAS with interval data type creates invalid parquet file
[ https://issues.apache.org/jira/browse/DRILL-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536907#comment-14536907 ] Mehant Baid commented on DRILL-3006: [~vicky] You are right, closing this as duplicate. CTAS with interval data type creates invalid parquet file - Key: DRILL-3006 URL: https://issues.apache.org/jira/browse/DRILL-3006 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Reporter: Mehant Baid Assignee: Steven Phillips Used the below CTAS statement: create table t6 as select interval '10' day interval_day_col from cp.`employee.json` limit 1; When I query the table 't6' the following exception is encountered: Caused by: java.io.IOException: Failure while trying to get footer for file file:/tmp/t6/0_0_0.parquet at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.convertToIOException(FooterGatherer.java:120) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.getValue(TimedRunnable.java:67) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:136) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer.getFooters(FooterGatherer.java:95) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooterHelper(ParquetGroupScan.java:229) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan.access$000(ParquetGroupScan.java:79) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan$1.run(ParquetGroupScan.java:206) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan$1.run(ParquetGroupScan.java:204) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) ~[hadoop-common-2.4.1.jar:na] at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooter(ParquetGroupScan.java:204) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] ... 21 common frames omitted Caused by: java.lang.IllegalArgumentException: Invalid FIXED_LEN_BYTE_ARRAY length: 0 at parquet.Preconditions.checkArgument(Preconditions.java:50) ~[parquet-common-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$PrimitiveBuilder.build(Types.java:320) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$PrimitiveBuilder.build(Types.java:250) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$Builder.named(Types.java:228) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:640) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:601) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:529) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:480) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at org.apache.drill.exec.store.parquet.FooterGatherer.readFooter(FooterGatherer.java:161) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:115) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:102) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:47)
[jira] [Updated] (DRILL-3008) Canonicalize Option Names, update calls to use validators rather than names.
[ https://issues.apache.org/jira/browse/DRILL-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau updated DRILL-3008: -- Attachment: DRILL-3008.patch Attaching initial patch to work from. Canonicalize Option Names, update calls to use validators rather than names. Key: DRILL-3008 URL: https://issues.apache.org/jira/browse/DRILL-3008 Project: Apache Drill Issue Type: Bug Reporter: Jacques Nadeau Assignee: Sean Hsuan-Yi Chu Attachments: DRILL-3008.patch Clean up option usages before 1.0: - Update the option names to be consistent. - Always use type-safe validators rather than direct names in Drill core and testcode. - Update test framework to take validator and setting rather than string - Remove all string ALTER SESSION settings in Drill codebase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3008) Canonicalize Option Names, update calls to use validators rather than names.
Jacques Nadeau created DRILL-3008: - Summary: Canonicalize Option Names, update calls to use validators rather than names. Key: DRILL-3008 URL: https://issues.apache.org/jira/browse/DRILL-3008 Project: Apache Drill Issue Type: Bug Reporter: Jacques Nadeau Assignee: Sean Hsuan-Yi Chu Clean up option usages before 1.0: - Update the option names to be consistent. - Always use type-safe validators rather than direct names in Drill core and testcode. - Update test framework to take validator and setting rather than string - Remove all string ALTER SESSION settings in Drill codebase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-1927) Aggregation or Group-By on a CASE expression with IN subquery fails
[
https://issues.apache.org/jira/browse/DRILL-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu resolved DRILL-1927.
--
Resolution: Fixed
Issue was resolved in DRILL-2914 (Calcite update). Test cases are added with
the patch in DRILL-2966
Aggregation or Group-By on a CASE expression with IN subquery fails
---
Key: DRILL-1927
URL: https://issues.apache.org/jira/browse/DRILL-1927
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.7.0
Reporter: Aman Sinha
Assignee: Sean Hsuan-Yi Chu
Fix For: 1.2.0
The following queries fail within Calcite and depend on the fix for
CALCITE-516. I am creating a Drill JIRA such that we can add tests with data
verification.
{code}
select (case when emp.empno in (3) then 0 else 1 end)
from emp
group by (case when emp.empno in (3) then 0 else 1 end);
java.lang.AssertionError: Internal error: while converting CASE WHEN
`EMP`.`EMPNO` IN (3) THEN 0 ELSE 1 END
at org.eigenbase.util.Util.newInternal(Util.java:750)
at
org.eigenbase.sql2rel.ReflectiveConvertletTable$1.convertCall(ReflectiveConvertletTable.java:93)
at
org.eigenbase.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:52)
at
org.eigenbase.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4093)
at
org.eigenbase.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:1)
at org.eigenbase.sql.SqlCall.accept(SqlCall.java:125)
at
org.eigenbase.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:3988)
at
org.eigenbase.sql2rel.SqlToRelConverter$AggConverter.addGroupExpr(SqlToRelConverter.java:4314)
at
org.eigenbase.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2240)
at
org.eigenbase.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2191)
at
org.eigenbase.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:514)
at
org.eigenbase.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:474)
{code}
{code}
SELECT SUM(CASE WHEN empno IN (3) THEN 0 ELSE 1 END) FROM emp;
java.lang.AssertionError: Internal error: Identifier 'EMP.EMPNO' is not a
group expr
at org.apache.calcite.util.Util.newInternal(Util.java:727)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3307)
at
org.apache.calcite.sql2rel.SqlToRelConverter.access$6(SqlToRelConverter.java:3296)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4299)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:1)
at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:271)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4182)
at
org.apache.calcite.sql2rel.SqlToRelConverter.substituteSubquery(SqlToRelConverter.java:979)
at
org.apache.calcite.sql2rel.SqlToRelConverter.replaceSubqueries(SqlToRelConverter.java:937)
at
org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2685)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2525)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:608)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2966) HAVING clause with CASE statement with IN predicate causes assertion
[
https://issues.apache.org/jira/browse/DRILL-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu updated DRILL-2966:
-
Attachment: DRILL-2966.1.patch
HAVING clause with CASE statement with IN predicate causes assertion
Key: DRILL-2966
URL: https://issues.apache.org/jira/browse/DRILL-2966
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Aman Sinha
Assignee: Sean Hsuan-Yi Chu
Fix For: 1.0.0
Attachments: DRILL-2966.1.patch
The following query fails in sql-to-rel conversion:
{code}
select count(*) from emp
group by emp.deptno
having sum(case when emp.empno in (1, 2, 3) then emp.sal else 0 end)
between 1.0 and 2.0
{code}
{code}
java.lang.AssertionError: Internal error: while converting CASE WHEN
`EMP`.`EMPNO` IN (1, 2, 3) THEN `EMP`.`SAL` ELSE 0 END
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4058)
at
org.apache.calcite.sql2rel.StandardConvertletTable.convertCase(StandardConvertletTable.java:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.calcite.sql2rel.ReflectiveConvertletTable$1.convertCall(ReflectiveConvertletTable.java:87)
at
org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertCall(SqlNodeToRexConverterImpl.java:60)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4212)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3668)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
at
org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4105)
at
org.apache.calcite.sql2rel.SqlToRelConverter$AggConverter.visit(SqlToRelConverter.java:4483)
at
org.apache.calcite.sql2rel.SqlToRelConverter$AggConverter.visit(SqlToRelConverter.java:4329)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
at
org.apache.calcite.sql2rel.SqlToRelConverter$AggConverter.visit(SqlToRelConverter.java:4528)
at
org.apache.calcite.sql2rel.SqlToRelConverter$AggConverter.visit(SqlToRelConverter.java:4329)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:130)
at
org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:2573)
at
org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:2510)
{code}
This is dependent on CALCITE-694 .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3006) CTAS with interval data type creates invalid parquet file
[ https://issues.apache.org/jira/browse/DRILL-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536904#comment-14536904 ] Victoria Markman commented on DRILL-3006: - Mehant, I think it is a duplicate of drill-1980 CTAS with interval data type creates invalid parquet file - Key: DRILL-3006 URL: https://issues.apache.org/jira/browse/DRILL-3006 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Reporter: Mehant Baid Assignee: Steven Phillips Used the below CTAS statement: create table t6 as select interval '10' day interval_day_col from cp.`employee.json` limit 1; When I query the table 't6' the following exception is encountered: Caused by: java.io.IOException: Failure while trying to get footer for file file:/tmp/t6/0_0_0.parquet at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.convertToIOException(FooterGatherer.java:120) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.getValue(TimedRunnable.java:67) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:136) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer.getFooters(FooterGatherer.java:95) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooterHelper(ParquetGroupScan.java:229) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan.access$000(ParquetGroupScan.java:79) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan$1.run(ParquetGroupScan.java:206) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.ParquetGroupScan$1.run(ParquetGroupScan.java:204) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_67] at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_67] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) ~[hadoop-common-2.4.1.jar:na] at org.apache.drill.exec.store.parquet.ParquetGroupScan.readFooter(ParquetGroupScan.java:204) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] ... 21 common frames omitted Caused by: java.lang.IllegalArgumentException: Invalid FIXED_LEN_BYTE_ARRAY length: 0 at parquet.Preconditions.checkArgument(Preconditions.java:50) ~[parquet-common-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$PrimitiveBuilder.build(Types.java:320) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$PrimitiveBuilder.build(Types.java:250) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.schema.Types$Builder.named(Types.java:228) ~[parquet-column-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.buildChildren(ParquetMetadataConverter.java:640) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.fromParquetSchema(ParquetMetadataConverter.java:601) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:543) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:529) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:480) ~[parquet-hadoop-1.6.0rc3-drill-r0.3.jar:1.6.0rc3-drill-r0.3] at org.apache.drill.exec.store.parquet.FooterGatherer.readFooter(FooterGatherer.java:161) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:115) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.parquet.FooterGatherer$FooterReader.runInner(FooterGatherer.java:102) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:47)
[jira] [Created] (DRILL-3010) Convert bad command error messages into UserExceptions in SqlHandlers
Venki Korukanti created DRILL-3010:
--
Summary: Convert bad command error messages into UserExceptions in
SqlHandlers
Key: DRILL-3010
URL: https://issues.apache.org/jira/browse/DRILL-3010
Project: Apache Drill
Issue Type: Bug
Components: Metadata
Affects Versions: 0.8.0
Reporter: Venki Korukanti
Assignee: Venki Korukanti
Fix For: 1.0.0
Currently SqlHandlers such as CreateTable or ViewHandler send the error
messages as bad command records. Instead we should throw a UserException.
{code}
0: jdbc:drill:zk=local create table t1 as select * from cp.`region.json`;
+++
| ok | summary |
+++
| false | Unable to create table. Schema [dfs.default] is immutable. |
+++
1 row selected (0.103 seconds)
{code}
Instead it should be like:
{code}
0: jdbc:drill:zk=10.10.30.143:5181 create table t1 as select * from
cp.`region.json`;
Error: PARSE ERROR: Unable to create or drop tables/views. Schema [dfs.default]
is immutable.
[Error Id: 3a92d026-3df7-4e8b-8988-2300463fa00b on centos64-30146.qa.lab:31010]
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2552) ZK disconnect to foreman node results in hung query on client
[ https://issues.apache.org/jira/browse/DRILL-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536926#comment-14536926 ] Jacques Nadeau commented on DRILL-2552: --- [~inramana], can you determine whether this continue to occur after the fixes that went in this morning? ZK disconnect to foreman node results in hung query on client - Key: DRILL-2552 URL: https://issues.apache.org/jira/browse/DRILL-2552 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Affects Versions: 0.8.0 Reporter: Ramana Inukonda Nagaraj Assignee: Ramana Inukonda Nagaraj Fix For: 1.0.0 Steps taken to recreate: 1. Startup drillbits on multiple nodes. (They all come up and form a 8 node cluster) 2. Start executing a long running query. 3. Use TCPKILL to kill all connections between foreman node and zookeeper port 5181. Drill seems to detect the node as gone and cancels the query but there is no communication of this back to the client which is hanging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2425) Wrong results when identifier change cases within the same data file
[
https://issues.apache.org/jira/browse/DRILL-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Phillips resolved DRILL-2425.
Resolution: Duplicate
Wrong results when identifier change cases within the same data file
Key: DRILL-2425
URL: https://issues.apache.org/jira/browse/DRILL-2425
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types
Affects Versions: 0.8.0
Reporter: Chun Chang
Assignee: Steven Phillips
Priority: Critical
Fix For: 1.0.0
#Fri Mar 06 16:51:10 EST 2015
git.commit.id.abbrev=fb293ba
I have the following JSON file that one of the identifier change cases:
{code}
[root@qa-node120 md-83]# hadoop fs -cat
/drill/testdata/complex_type/json/schema/a.json
{SOURCE: ebm,msAddressIpv6Array: null}
{SOURCE: ebm,msAddressIpv6Array: {msAddressIpv6_1:99.111.222.0,
msAddressIpv6_2:88.222.333.0}}
{SOURCE: ebm,msAddressIpv6Array: {msAddressIpv6_1:99.111.222.1,
msAddressIpv6_2:88.222.333.1}}
{SOURCE: ebm,msAddressIpv6Array: {msaddressipv6_1:99.111.222.2,
msAddressIpv6_2:88.222.333.2}}
{code}
Query this file through drill gives wrong results:
{code}
0: jdbc:drill:schema=dfs.drillTestDirComplexJ select
t.msAddressIpv6Array.msAddressIpv6_1 as msAddressIpv6_1 from `schema/a.json`
t;
+-+
| msAddressIpv6_1 |
+-+
| null|
| null|
| null|
| 99.111.222.2|
+-+
{code}
plan:
{code}
0: jdbc:drill:schema=dfs.drillTestDirComplexJ explain plan for select
t.msAddressIpv6Array.msAddressIpv6_1 as msAddressIpv6_1 from `schema/a.json`
t;
+++
|text|json|
+++
| 00-00Screen
00-01 Project(msAddressIpv6_1=[ITEM($0, 'msAddressIpv6_1')])
00-02Scan(groupscan=[EasyGroupScan
[selectionRoot=/drill/testdata/complex_type/json/schema/a.json, numFiles=1,
columns=[`msAddressIpv6Array`.`msAddressIpv6_1`],
files=[maprfs:/drill/testdata/complex_type/json/schema/a.json]]])
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2086) mapr profile - use MapR 4.0.2
[ https://issues.apache.org/jira/browse/DRILL-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-2086: --- Fix Version/s: (was: 1.0.0) 1.1.0 mapr profile - use MapR 4.0.2 - Key: DRILL-2086 URL: https://issues.apache.org/jira/browse/DRILL-2086 Project: Apache Drill Issue Type: Improvement Components: Tools, Build Test Reporter: Patrick Wong Assignee: Steven Phillips Fix For: 1.1.0 Attachments: DRILL-2086.1.patch.txt This will greatly simplify some other things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3009) Reduce the IN list threshold to 10 to take advantage of Values operator
[ https://issues.apache.org/jira/browse/DRILL-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Sinha updated DRILL-3009: -- Assignee: Jinfeng Ni (was: Aman Sinha) Reduce the IN list threshold to 10 to take advantage of Values operator --- Key: DRILL-3009 URL: https://issues.apache.org/jira/browse/DRILL-3009 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Reporter: Aman Sinha Assignee: Jinfeng Ni Fix For: 1.0.0 Attachments: 0001-DRILL-3009-Lower-IN-list-threshold-change-is-in-Calc.patch The IN-list threshold is currently set to 200 (it was bumped up from 20 previously since Drill did not have the Values operator). Now that Drill can support large IN lists through the Values operator, we should drop the threshold back and in fact lower it to 10. For lists below this threshold, Calcite will build a binary tree of OR's. Above this threshold, it will create a ValuesRel . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3009) Reduce the IN list threshold to 10 to take advantage of Values operator
[ https://issues.apache.org/jira/browse/DRILL-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Sinha updated DRILL-3009: -- Attachment: 0001-DRILL-3009-Lower-IN-list-threshold-change-is-in-Calc.patch Changed the threshold in Drill's Calcite branch. Bumped up the version to 1.1.0-drill-r6. [~jni] could you review the simple patch ? Reduce the IN list threshold to 10 to take advantage of Values operator --- Key: DRILL-3009 URL: https://issues.apache.org/jira/browse/DRILL-3009 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Reporter: Aman Sinha Assignee: Aman Sinha Fix For: 1.0.0 Attachments: 0001-DRILL-3009-Lower-IN-list-threshold-change-is-in-Calc.patch The IN-list threshold is currently set to 200 (it was bumped up from 20 previously since Drill did not have the Values operator). Now that Drill can support large IN lists through the Values operator, we should drop the threshold back and in fact lower it to 10. For lists below this threshold, Calcite will build a binary tree of OR's. Above this threshold, it will create a ValuesRel . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3009) Reduce the IN list threshold to 10 to take advantage of Values operator
[ https://issues.apache.org/jira/browse/DRILL-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536982#comment-14536982 ] Jacques Nadeau commented on DRILL-3009: --- LGTM +1 Reduce the IN list threshold to 10 to take advantage of Values operator --- Key: DRILL-3009 URL: https://issues.apache.org/jira/browse/DRILL-3009 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Reporter: Aman Sinha Assignee: Jinfeng Ni Fix For: 1.0.0 Attachments: 0001-DRILL-3009-Lower-IN-list-threshold-change-is-in-Calc.patch The IN-list threshold is currently set to 200 (it was bumped up from 20 previously since Drill did not have the Values operator). Now that Drill can support large IN lists through the Values operator, we should drop the threshold back and in fact lower it to 10. For lists below this threshold, Calcite will build a binary tree of OR's. Above this threshold, it will create a ValuesRel . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2304) Case sensitivity - system and session options are case sensitive
[
https://issues.apache.org/jira/browse/DRILL-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam updated DRILL-2304:
---
Assignee: Jason Altekruse (was: Sudheesh Katkam)
Case sensitivity - system and session options are case sensitive
Key: DRILL-2304
URL: https://issues.apache.org/jira/browse/DRILL-2304
Project: Apache Drill
Issue Type: Bug
Components: Storage - Information Schema
Affects Versions: 0.8.0
Reporter: Ramana Inukonda Nagaraj
Assignee: Jason Altekruse
Priority: Minor
Fix For: 1.2.0
Attachments: DRILL-2304.1.patch.txt, DRILL-2304.2.patch.txt
TBH I am not sure if this is a bug.
When trying to set a session option and I specify the name in a different
case the alter command fails. Considering the way we store session options
this might be an invalid bug but considering how typical Database hints and
options work this is a bug.
{code}
0: jdbc:drill: alter SESSION set `STORE.PARQUET.COMPRESSION`='GZIP';
Query failed: SetOptionException: Unknown option: STORE.PARQUET.COMPRESSION
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2977) In WorkManager, startFragmentPendingRemote() and addFragmentRunner() need to be permuted
[
https://issues.apache.org/jira/browse/DRILL-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudheesh Katkam updated DRILL-2977:
---
Assignee: Venki Korukanti (was: Sudheesh Katkam)
In WorkManager, startFragmentPendingRemote() and addFragmentRunner() need to
be permuted
Key: DRILL-2977
URL: https://issues.apache.org/jira/browse/DRILL-2977
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Reporter: Deneche A. Hakim
Assignee: Venki Korukanti
Fix For: 1.0.0
Attachments: DRILL-2977.2.patch.txt
In WorkManager 2 methods can be used to start a fragment executor:
- startFragmentPendingRemote(FragmentManager) will start running a fragment
by calling executor.execute(fragment)
- addFragmentRunner(FragmentExecutor) will add the fragment to
runningFragments and start a fragment after wrapping it inside a
SelfCleanableRunner to make sure it's manager is removed from the
WorkEventBus once finished.
Looking at how both methods are called it seems that we are actually calling
startFragmentPendingRemote() for fragments that were added to the
WorkEventBus and we call addFragmentRunner() for fragment that were not added
to the workBus!!! For example in Foreman.setupRootFragment() we have the
following:
{code}
if (buffers.isDone()) {
// if we don't have to wait for any incoming data, start the fragment
runner.
bee.addFragmentRunner(fragmentManager.getRunnable());
} else {
// if we do, record the fragment manager in the workBus.
drillbitContext.getWorkBus().addFragmentManager(fragmentManager);
}
{code}
The names of the methods are correct, but we need to switch their
implementations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-2778) Killing the drillbit which is the foreman results in hung sqlline
[ https://issues.apache.org/jira/browse/DRILL-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau resolved DRILL-2778. --- Resolution: Fixed Resolved in 960f876 Killing the drillbit which is the foreman results in hung sqlline - Key: DRILL-2778 URL: https://issues.apache.org/jira/browse/DRILL-2778 Project: Apache Drill Issue Type: Bug Components: Execution - Flow Reporter: Ramana Inukonda Nagaraj Assignee: Jacques Nadeau Fix For: 1.0.0 Killed one of the drillbits which is the foreman for the query- Profiles page reports that query has cancelled. sqlline hangs though- Does not look like the cancel went all the way through. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-2886) JDBC driver doesn't detect lost connection
[ https://issues.apache.org/jira/browse/DRILL-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau resolved DRILL-2886. --- Resolution: Fixed Resolved in 960f876 JDBC driver doesn't detect lost connection -- Key: DRILL-2886 URL: https://issues.apache.org/jira/browse/DRILL-2886 Project: Apache Drill Issue Type: Bug Components: Client - JDBC Reporter: Jacques Nadeau Assignee: Jacques Nadeau Priority: Critical Fix For: 1.0.0 Open a sqlline session. Terminate the Drillbit you attached to. Run a query. The experience is that the query doesn't fail, just hangs indefinitely. Correct response is connection closed or similar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3008) Canonicalize Option Names, update calls to use validators rather than names.
[ https://issues.apache.org/jira/browse/DRILL-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536969#comment-14536969 ] Jacques Nadeau commented on DRILL-3008: --- Another note, please make sure the options iterator for the SystemTable.OPTIONS is alphabetically ordered. Canonicalize Option Names, update calls to use validators rather than names. Key: DRILL-3008 URL: https://issues.apache.org/jira/browse/DRILL-3008 Project: Apache Drill Issue Type: Bug Reporter: Jacques Nadeau Assignee: Sean Hsuan-Yi Chu Attachments: DRILL-3008.patch Clean up option usages before 1.0: - Update the option names to be consistent. - Always use type-safe validators rather than direct names in Drill core and testcode. - Update test framework to take validator and setting rather than string - Remove all string ALTER SESSION settings in Drill codebase -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3009) Reduce the IN list threshold to 10 to take advantage of Values operator
Aman Sinha created DRILL-3009: - Summary: Reduce the IN list threshold to 10 to take advantage of Values operator Key: DRILL-3009 URL: https://issues.apache.org/jira/browse/DRILL-3009 Project: Apache Drill Issue Type: Bug Components: Query Planning Optimization Reporter: Aman Sinha Assignee: Aman Sinha The IN-list threshold is currently set to 200 (it was bumped up from 20 previously since Drill did not have the Values operator). Now that Drill can support large IN lists through the Values operator, we should drop the threshold back and in fact lower it to 10. For lists below this threshold, Calcite will build a binary tree of OR's. Above this threshold, it will create a ValuesRel . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1942) Improve off-heap memory usage tracking
[ https://issues.apache.org/jira/browse/DRILL-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Westin updated DRILL-1942: Attachment: DRILL-1942.3.patch.txt Improve off-heap memory usage tracking -- Key: DRILL-1942 URL: https://issues.apache.org/jira/browse/DRILL-1942 Project: Apache Drill Issue Type: Improvement Components: Execution - Relational Operators Reporter: Chris Westin Assignee: Chris Westin Fix For: 1.0.0 Attachments: DRILL-1942.1.patch.txt, DRILL-1942.2.patch.txt, DRILL-1942.3.patch.txt We're using a lot more memory than we think we should. We may be leaking it, or not releasing it as soon as we could. This is a call to come up with some improved tracking so that we can get statistics out about exactly where we're using it, and whether or not we can release it earlier. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2914) regression: Mondrian query534.q, drill give wrong result
[
https://issues.apache.org/jira/browse/DRILL-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu updated DRILL-2914:
-
Attachment: DRILL-2914.1.patch
regression: Mondrian query534.q, drill give wrong result
Key: DRILL-2914
URL: https://issues.apache.org/jira/browse/DRILL-2914
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Chun Chang
Assignee: Sean Hsuan-Yi Chu
Priority: Critical
Fix For: 1.0.0
Attachments: DRILL-2914.1.patch, customer
#Wed Apr 29 14:39:22 EDT 2015
git.commit.id.abbrev=f5b0f49
Running mondrian tests and noticed several regressions. This one is related
to group by with select * subquery.
{code}
SELECT gender2.gender AS c0
FROM (SELECT *
FROM customer) AS gender2
GROUP BY gender2.gender
ORDER BY gender2.gender ASC nulls last;
{code}
Correct result from postgres:
{code}
foodmart=# select gender2.gender as c0 from (SELECT * FROM customer) as
gender2 group by gender2.gender order by gender2.gender ASC NULLS LAST;
c0
F
M
(2 rows)
{code}
Wrong result from drill:
{code}
0: jdbc:drill:schema=dfs.drillTestDirAdvanced select gender2.gender as c0
from (SELECT * FROM customer) as gender2 group by gender2.gender order by
gender2.gender ASC NULLS LAST;
++
| c0 |
++
| null |
++
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2941) Update RPC layer to avoid writing local data messages to socket
[ https://issues.apache.org/jira/browse/DRILL-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-2941: --- Assignee: Jacques Nadeau (was: Steven Phillips) Update RPC layer to avoid writing local data messages to socket --- Key: DRILL-2941 URL: https://issues.apache.org/jira/browse/DRILL-2941 Project: Apache Drill Issue Type: Bug Components: Execution - RPC Reporter: Jacques Nadeau Assignee: Jacques Nadeau Fix For: 1.0.0 Attachments: DRILL-2941.patch, DRILL-2941.patch Right now, if we send a fragment record batch to localhost, we still traverse the RPC layer. We should short-circuit this path. This is especially important in light of the mux and demux exchanges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2971) If BitBit connection is unexpectedly closed and we were already blocked on writing to socket, we'll stay forever in ResettableBarrier.await()
[ https://issues.apache.org/jira/browse/DRILL-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-2971: --- Assignee: Jacques Nadeau (was: Steven Phillips) If BitBit connection is unexpectedly closed and we were already blocked on writing to socket, we'll stay forever in ResettableBarrier.await() --- Key: DRILL-2971 URL: https://issues.apache.org/jira/browse/DRILL-2971 Project: Apache Drill Issue Type: Bug Components: Execution - RPC Reporter: Jacques Nadeau Assignee: Jacques Nadeau Fix For: 1.0.0 Attachments: DRILL-2971.patch We need to reset the ResettableBarrier if the connection dies so that the message can be failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2941) Update RPC layer to avoid writing local data messages to socket
[ https://issues.apache.org/jira/browse/DRILL-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536302#comment-14536302 ] Steven Phillips commented on DRILL-2941: +1 Update RPC layer to avoid writing local data messages to socket --- Key: DRILL-2941 URL: https://issues.apache.org/jira/browse/DRILL-2941 Project: Apache Drill Issue Type: Bug Components: Execution - RPC Reporter: Jacques Nadeau Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2941.patch, DRILL-2941.patch Right now, if we send a fragment record batch to localhost, we still traverse the RPC layer. We should short-circuit this path. This is especially important in light of the mux and demux exchanges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2971) If BitBit connection is unexpectedly closed and we were already blocked on writing to socket, we'll stay forever in ResettableBarrier.await()
[ https://issues.apache.org/jira/browse/DRILL-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536322#comment-14536322 ] Steven Phillips commented on DRILL-2971: +1 If BitBit connection is unexpectedly closed and we were already blocked on writing to socket, we'll stay forever in ResettableBarrier.await() --- Key: DRILL-2971 URL: https://issues.apache.org/jira/browse/DRILL-2971 Project: Apache Drill Issue Type: Bug Components: Execution - RPC Reporter: Jacques Nadeau Assignee: Steven Phillips Fix For: 1.0.0 Attachments: DRILL-2971.patch We need to reset the ResettableBarrier if the connection dies so that the message can be failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2743) Parquet file metadata caching
[ https://issues.apache.org/jira/browse/DRILL-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-2743: --- Fix Version/s: (was: 1.0.0) 1.1.0 Parquet file metadata caching - Key: DRILL-2743 URL: https://issues.apache.org/jira/browse/DRILL-2743 Project: Apache Drill Issue Type: New Feature Components: Storage - Parquet Reporter: Steven Phillips Assignee: Steven Phillips Fix For: 1.1.0 Attachments: DRILL-2743.patch, drill.parquet_metadata To run a query against parquet files, we have to first recursively search the directory tree for all of the files, get the block locations for each file, and read the footer from each file, and this is done during the planning phase. When there are many files, this can result in a very large delay in running the query, and it does not scale. However, there isn't really any need to read the footers during planning, if we instead treat each parquet file as a single work unit, all we need to know are the block locations for the file, the number of rows, and the columns. We should store only the information which we need for planning in a file located in the top directory for a given parquet table, and then we can delay reading of the footers until execution time, which can be done in parallel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2476) Handle IterOutcome.STOP in buildSchema()
[
https://issues.apache.org/jira/browse/DRILL-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536327#comment-14536327
]
Steven Phillips commented on DRILL-2476:
+1
Handle IterOutcome.STOP in buildSchema()
Key: DRILL-2476
URL: https://issues.apache.org/jira/browse/DRILL-2476
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 0.7.0
Reporter: Sudheesh Katkam
Assignee: Steven Phillips
Fix For: 1.0.0
Attachments: DRILL-2476.1.patch.txt
There are some {{RecordBatch}} implementations like {{HashAggBatch}} that
override the {{buildSchema()}} function. The overriding functions do not
handle {{IterOutcome.STOP}}. This causes the {{FragmentContext}} to receive
two failures in some cases (linked JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2915) Regression: Mondrian query5614.q - Query failed: SYSTEM ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join
[
https://issues.apache.org/jira/browse/DRILL-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu updated DRILL-2915:
-
Attachment: DRILL-2915.1.patch
Regression: Mondrian query5614.q - Query failed: SYSTEM ERROR: This query
cannot be planned possibly due to either a cartesian join or an inequality
join
-
Key: DRILL-2915
URL: https://issues.apache.org/jira/browse/DRILL-2915
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Chun Chang
Assignee: Aman Sinha
Priority: Critical
Fix For: 1.1.0
Attachments: DRILL-2915.1.patch, mondrian_query5614.explain
#Wed Apr 29 14:39:22 EDT 2015
git.commit.id.abbrev=f5b0f49
The following mondrian query fails now.
{code}
SELECT store.store_state AS c0,
Count(DISTINCT sales_fact_1997.customer_id) AS m0
FROM store AS store,
sales_fact_1997 AS sales_fact_1997,
time_by_day AS time_by_day,
product_class AS product_class,
product AS product
WHERE sales_fact_1997.store_id = store.store_id
AND store.store_state = 'CA'
AND sales_fact_1997.time_id = time_by_day.time_id
AND sales_fact_1997.product_id = product.product_id
AND product.product_class_id = product_class.product_class_id
AND ( ( product_class.product_family = 'Food'
AND time_by_day.quarter = 'Q1'
AND time_by_day.the_year = 1997 )
OR ( product_class.product_family = 'Drink'
AND time_by_day.month_of_year = 4
AND time_by_day.quarter = 'Q2'
AND time_by_day.the_year = 1997 ) )
GROUP BY store.store_state;
{code}
postgres:
{code}
foodmart=# select store.store_state as c0, count(distinct
sales_fact_1997.customer_id) as m0 from store as store, sales_fact_1997 as
sales_fact_1997, time_by_day as time_by_day, product_class as product_class,
product as product where sales_fact_1997.store_id = store.store_id and
store.store_state = 'CA' and sales_fact_1997.time_id = time_by_day.time_id
and sales_fact_1997.product_id = product.product_id and
product.product_class_id = product_class.product_class_id and
((product_class.product_family = 'Food' and time_by_day.quarter = 'Q1' and
time_by_day.the_year = 1997) or (product_class.product_family = 'Drink' and
time_by_day.month_of_year = 4 and time_by_day.quarter = 'Q2' and
time_by_day.the_year = 1997)) group by store.store_state;
c0 | m0
+--
CA | 1175
(1 row)
{code}
drill failed
{code}
0: jdbc:drill:schema=dfs.drillTestDirAdvanced select store.store_state as
c0, count(distinct sales_fact_1997.customer_id) as m0 from store as store,
sales_fact_1997 as sales_fact_1997, time_by_day as time_by_day, product_class
as product_class, product as product where sales_fact_1997.store_id =
store.store_id and store.store_state = 'CA' and sales_fact_1997.time_id =
time_by_day.time_id and sales_fact_1997.product_id = product.product_id and
product.product_class_id = product_class.product_class_id and
((product_class.product_family = 'Food' and time_by_day.quarter = 'Q1' and
time_by_day.the_year = 1997) or (product_class.product_family = 'Drink' and
time_by_day.month_of_year = 4 and time_by_day.quarter = 'Q2' and
time_by_day.the_year = 1997)) group by store.store_state;
Query failed: SYSTEM ERROR: This query cannot be planned possibly due to
either a cartesian join or an inequality join
[3eb99963-92aa-4129-844f-fe43839537b9 on qa-node119.qa.lab:31010]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2915) Regression: Mondrian query5614.q - Query failed: SYSTEM ERROR: This query cannot be planned possibly due to either a cartesian join or an inequality join
[
https://issues.apache.org/jira/browse/DRILL-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu updated DRILL-2915:
-
Attachment: (was: DRILL-2915.1.patch)
Regression: Mondrian query5614.q - Query failed: SYSTEM ERROR: This query
cannot be planned possibly due to either a cartesian join or an inequality
join
-
Key: DRILL-2915
URL: https://issues.apache.org/jira/browse/DRILL-2915
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Chun Chang
Assignee: Aman Sinha
Priority: Critical
Fix For: 1.1.0
Attachments: mondrian_query5614.explain
#Wed Apr 29 14:39:22 EDT 2015
git.commit.id.abbrev=f5b0f49
The following mondrian query fails now.
{code}
SELECT store.store_state AS c0,
Count(DISTINCT sales_fact_1997.customer_id) AS m0
FROM store AS store,
sales_fact_1997 AS sales_fact_1997,
time_by_day AS time_by_day,
product_class AS product_class,
product AS product
WHERE sales_fact_1997.store_id = store.store_id
AND store.store_state = 'CA'
AND sales_fact_1997.time_id = time_by_day.time_id
AND sales_fact_1997.product_id = product.product_id
AND product.product_class_id = product_class.product_class_id
AND ( ( product_class.product_family = 'Food'
AND time_by_day.quarter = 'Q1'
AND time_by_day.the_year = 1997 )
OR ( product_class.product_family = 'Drink'
AND time_by_day.month_of_year = 4
AND time_by_day.quarter = 'Q2'
AND time_by_day.the_year = 1997 ) )
GROUP BY store.store_state;
{code}
postgres:
{code}
foodmart=# select store.store_state as c0, count(distinct
sales_fact_1997.customer_id) as m0 from store as store, sales_fact_1997 as
sales_fact_1997, time_by_day as time_by_day, product_class as product_class,
product as product where sales_fact_1997.store_id = store.store_id and
store.store_state = 'CA' and sales_fact_1997.time_id = time_by_day.time_id
and sales_fact_1997.product_id = product.product_id and
product.product_class_id = product_class.product_class_id and
((product_class.product_family = 'Food' and time_by_day.quarter = 'Q1' and
time_by_day.the_year = 1997) or (product_class.product_family = 'Drink' and
time_by_day.month_of_year = 4 and time_by_day.quarter = 'Q2' and
time_by_day.the_year = 1997)) group by store.store_state;
c0 | m0
+--
CA | 1175
(1 row)
{code}
drill failed
{code}
0: jdbc:drill:schema=dfs.drillTestDirAdvanced select store.store_state as
c0, count(distinct sales_fact_1997.customer_id) as m0 from store as store,
sales_fact_1997 as sales_fact_1997, time_by_day as time_by_day, product_class
as product_class, product as product where sales_fact_1997.store_id =
store.store_id and store.store_state = 'CA' and sales_fact_1997.time_id =
time_by_day.time_id and sales_fact_1997.product_id = product.product_id and
product.product_class_id = product_class.product_class_id and
((product_class.product_family = 'Food' and time_by_day.quarter = 'Q1' and
time_by_day.the_year = 1997) or (product_class.product_family = 'Drink' and
time_by_day.month_of_year = 4 and time_by_day.quarter = 'Q2' and
time_by_day.the_year = 1997)) group by store.store_state;
Query failed: SYSTEM ERROR: This query cannot be planned possibly due to
either a cartesian join or an inequality join
[3eb99963-92aa-4129-844f-fe43839537b9 on qa-node119.qa.lab:31010]
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-2914) regression: Mondrian query534.q, drill give wrong result
[
https://issues.apache.org/jira/browse/DRILL-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aman Sinha resolved DRILL-2914.
---
Resolution: Fixed
Committed [~seanhychu]'s patch in a8c938767.
regression: Mondrian query534.q, drill give wrong result
Key: DRILL-2914
URL: https://issues.apache.org/jira/browse/DRILL-2914
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Chun Chang
Assignee: Sean Hsuan-Yi Chu
Priority: Critical
Fix For: 1.0.0
Attachments: DRILL-2914.1.patch, customer
#Wed Apr 29 14:39:22 EDT 2015
git.commit.id.abbrev=f5b0f49
Running mondrian tests and noticed several regressions. This one is related
to group by with select * subquery.
{code}
SELECT gender2.gender AS c0
FROM (SELECT *
FROM customer) AS gender2
GROUP BY gender2.gender
ORDER BY gender2.gender ASC nulls last;
{code}
Correct result from postgres:
{code}
foodmart=# select gender2.gender as c0 from (SELECT * FROM customer) as
gender2 group by gender2.gender order by gender2.gender ASC NULLS LAST;
c0
F
M
(2 rows)
{code}
Wrong result from drill:
{code}
0: jdbc:drill:schema=dfs.drillTestDirAdvanced select gender2.gender as c0
from (SELECT * FROM customer) as gender2 group by gender2.gender order by
gender2.gender ASC NULLS LAST;
++
| c0 |
++
| null |
++
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2941) Update RPC layer to avoid writing local data messages to socket
[ https://issues.apache.org/jira/browse/DRILL-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacques Nadeau updated DRILL-2941: -- Attachment: DRILL-2941.patch Added reviewboard https://reviews.apache.org/r/34013/ Update RPC layer to avoid writing local data messages to socket --- Key: DRILL-2941 URL: https://issues.apache.org/jira/browse/DRILL-2941 Project: Apache Drill Issue Type: Bug Components: Execution - RPC Reporter: Jacques Nadeau Assignee: Jacques Nadeau Fix For: 1.0.0 Attachments: DRILL-2941.patch, DRILL-2941.patch, DRILL-2941.patch Right now, if we send a fragment record batch to localhost, we still traverse the RPC layer. We should short-circuit this path. This is especially important in light of the mux and demux exchanges. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2914) regression: Mondrian query534.q, drill give wrong result
[
https://issues.apache.org/jira/browse/DRILL-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536748#comment-14536748
]
Aman Sinha commented on DRILL-2914:
---
+1. I have reviewed the 1.1.0-drill-r5 patch on Calcite side but for future
reference pls post it here.
regression: Mondrian query534.q, drill give wrong result
Key: DRILL-2914
URL: https://issues.apache.org/jira/browse/DRILL-2914
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Reporter: Chun Chang
Assignee: Sean Hsuan-Yi Chu
Priority: Critical
Fix For: 1.0.0
Attachments: DRILL-2914.1.patch, customer
#Wed Apr 29 14:39:22 EDT 2015
git.commit.id.abbrev=f5b0f49
Running mondrian tests and noticed several regressions. This one is related
to group by with select * subquery.
{code}
SELECT gender2.gender AS c0
FROM (SELECT *
FROM customer) AS gender2
GROUP BY gender2.gender
ORDER BY gender2.gender ASC nulls last;
{code}
Correct result from postgres:
{code}
foodmart=# select gender2.gender as c0 from (SELECT * FROM customer) as
gender2 group by gender2.gender order by gender2.gender ASC NULLS LAST;
c0
F
M
(2 rows)
{code}
Wrong result from drill:
{code}
0: jdbc:drill:schema=dfs.drillTestDirAdvanced select gender2.gender as c0
from (SELECT * FROM customer) as gender2 group by gender2.gender order by
gender2.gender ASC NULLS LAST;
++
| c0 |
++
| null |
++
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3004) Failure in planning join when disabling hash join and exchanges
Jacques Nadeau created DRILL-3004:
-
Summary: Failure in planning join when disabling hash join and
exchanges
Key: DRILL-3004
URL: https://issues.apache.org/jira/browse/DRILL-3004
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Reporter: Jacques Nadeau
Assignee: Jinfeng Ni
Fix For: 1.1.0
{code}
ALTER SESSION SET `planner.enable_hashjoin` = false;
{code}
{code}
ALTER SESSION SET `planner.disable_exchanges` = true;
{code}
{code}
SELECT
nations.N_NAME,
regions.R_NAME
FROM
cp.`tpch/nation.parquet` nations
JOIN
cp.`tpch/region.parquet` regions
on nations.N_REGIONKEY = regions.R_REGIONKEY where 1 = 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2074) Queries fail with OutOfMemory Exception when Hash Join Agg are turned off
[
https://issues.apache.org/jira/browse/DRILL-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2074:
Assignee: Deneche A. Hakim (was: Chris Westin)
Queries fail with OutOfMemory Exception when Hash Join Agg are turned off
---
Key: DRILL-2074
URL: https://issues.apache.org/jira/browse/DRILL-2074
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Abhishek Girish
Assignee: Deneche A. Hakim
Fix For: 1.0.0
Attachments: 05_par1000.q, 05_par1000_d6e54ab.logical.plan,
05_par1000_d6e54ab.verbose.plan, drill-env.sh
Query attached.
Hash Join and Hash Agg were turned off. And the following property was added
to drill-override.conf:
sort: {
purge.threshold : 100,
external: {
batch.size : 4000,
spill: {
batch.size : 4000,
group.size : 100,
threshold : 200,
directories : [ /drill_spill ],
fs : maprfs:///
}
}
}
Query failed with the below error message:
Query failed: RemoteRpcException: Failure while running fragment., Unable to
allocate sv2 buffer after repeated attempts [
faf3044a-e14a-427b-b66d-7bcd7522ead5 on drone-42:31010 ]
[ faf3044a-e14a-427b-b66d-7bcd7522ead5 on drone-42:31010 ]
Log Snippets:
2015-01-26 20:07:33,239 atsqa8c42.qa.lab
[2b396307-2c1e-3486-90bc-fbaf09fbeb3e:frag:15:51] ERROR
o.a.d.e.w.f.AbstractStatusReporter - Error
faf3044a-e14a-427b-b66d-7bcd7522ead5: Failure while running fragment.
java.lang.RuntimeException:
org.apache.drill.exec.memory.OutOfMemoryException: Unable to allocate sv2
buffer after repeated attempts
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:309)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.JoinStatus.nextRight(JoinStatus.java:80)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:95)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.MergeJoinBatch.innerNext(MergeJoinBatch.java:147)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
[jira] [Resolved] (DRILL-2960) Default hive storage plugin missing from fresh drill install
[ https://issues.apache.org/jira/browse/DRILL-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Altekruse resolved DRILL-2960. Resolution: Fixed Fixed in d4f9bf2e994969c863b2b90b58e90139d242b106 Default hive storage plugin missing from fresh drill install Key: DRILL-2960 URL: https://issues.apache.org/jira/browse/DRILL-2960 Project: Apache Drill Issue Type: Bug Components: Storage - Hive Affects Versions: 0.9.0 Reporter: Krystal Assignee: Jason Altekruse Fix For: 1.0.0 Attachments: 2960.patch Installed drill on a fresh node. The default storage plugin for hive is missing from the webUI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2281) Drill never returns when we use aggregate functions after a join with an order by
[
https://issues.apache.org/jira/browse/DRILL-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Westin updated DRILL-2281:
Assignee: Deneche A. Hakim (was: Chris Westin)
Drill never returns when we use aggregate functions after a join with an
order by
-
Key: DRILL-2281
URL: https://issues.apache.org/jira/browse/DRILL-2281
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Deneche A. Hakim
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=6676f2d
The below query never returns : (Order by seems to be the culprit)
{code}
create view v1 as select uid, flatten(events) event from `data.json`;
create view v2 as select uid, flatten(transactions) transaction from
`data.json`;
select v1.uid, MAX(v2.transaction.amount), MIN(v1.event.event_time) from v1
inner join v2 on v1.uid = v2.uid where v2.transaction.trans_time 0 group by
v1.uid order by v1.uid;
{code}
There seems to be constant activity in the drillbit.log file. The below
message is continuously displayed in the log file
{code}
2015-02-20 23:35:04,450 [2b183b65-4551-bb9a-35ca-b71b9eedc4d6:frag:1:2] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
2015-02-20 23:35:04,451 [2b183b65-4551-bb9a-35ca-b71b9eedc4d6:frag:1:2] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
2015-02-20 23:35:04,451 [2b183b65-4551-bb9a-35ca-b71b9eedc4d6:frag:1:2] INFO
o.a.d.exec.vector.BaseValueVector - Realloc vector null. [32768] - [65536]
{code}
Drill returns correct data when we remove one of the agg functions or use
multiple aggs from the same side of the join. The below queries work :
{code}
select v1.uid, MAX(v2.transaction.amount) from v1 inner join v2 on v1.uid =
v2.uid where v2.transaction.trans_time 0 group by v1.uid order by v1.uid;
select v1.uid, MAX(v2.transaction.amount), MAX(v2.transaction.amount) from
v1 inner join v2 on v1.uid = v2.uid where v2.transaction.trans_time 0 group
by v1.uid order by v1.uid;
{code}
Attached the dataset which contains 2 records. I copied over the same 2
records 5 times and ran the queries on the data set. Let me know if you
need anything else.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2946) Tableau 9.0 Desktop Enablement Document
[ https://issues.apache.org/jira/browse/DRILL-2946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536788#comment-14536788 ] Bob Rumsby commented on DRILL-2946: --- This document is published here: http://tshiran.github.io/drill/docs/using-apache-drill-with-tableau-9-desktop/ I need to fix a couple of broken links before pushing it to the public site. I will add the Tableau Server doc on Monday. Tableau 9.0 Desktop Enablement Document --- Key: DRILL-2946 URL: https://issues.apache.org/jira/browse/DRILL-2946 Project: Apache Drill Issue Type: Improvement Components: Documentation Affects Versions: 0.9.0 Reporter: Andries Engelbrecht Assignee: Bridget Bevens Attachments: Tableau 9 Desktop Drill Configuration.docx Documentation for Tableau 9.0 Desktop enablement. Includes authentication with Drill 0.9 and later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3005) Spurious Error messages when using PrintingResultsListener
Jacques Nadeau created DRILL-3005: - Summary: Spurious Error messages when using PrintingResultsListener Key: DRILL-3005 URL: https://issues.apache.org/jira/browse/DRILL-3005 Project: Apache Drill Issue Type: Bug Reporter: Jacques Nadeau Fix For: 1.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2697) Pause injections should pause indefinitely until signalled
[ https://issues.apache.org/jira/browse/DRILL-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudheesh Katkam updated DRILL-2697: --- Attachment: DRILL-2697.5.patch.txt Pause injections should pause indefinitely until signalled -- Key: DRILL-2697 URL: https://issues.apache.org/jira/browse/DRILL-2697 Project: Apache Drill Issue Type: Improvement Components: Execution - Flow Affects Versions: 0.9.0 Reporter: Sudheesh Katkam Assignee: Chris Westin Fix For: 1.0.0 Attachments: DRILL-2697.1.patch.txt, DRILL-2697.5.patch.txt Currently injected pauses make threads sleep for a specified time. This can be an enhanced to stop the thread indefinitely using a CountDownLatch. It is quite similar to how cancellation works. Tasks: (a) Add another message to RPC layer to signal paused remote threads to resume (through ControlHandler) by counting down. Complications if the thread has not reached the pause site yet. (b) Add resume signal (like ctrl-c) to sqlline (further enhancement: another signal to trigger pause from sqlline) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2757) Verify operators correctly handle low memory conditions and cancellations
[ https://issues.apache.org/jira/browse/DRILL-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2757: Attachment: DRILL-2757.3.patch.txt DRILL-2757.2.patch.txt Verify operators correctly handle low memory conditions and cancellations - Key: DRILL-2757 URL: https://issues.apache.org/jira/browse/DRILL-2757 Project: Apache Drill Issue Type: Bug Components: Execution - Flow, Execution - Relational Operators Affects Versions: 0.8.0 Reporter: Chris Westin Assignee: Steven Phillips Priority: Blocker Fix For: 1.0.0 Attachments: DRILL-2757.1.patch.txt, DRILL-2757.2.patch.txt, DRILL-2757.3.patch.txt Check that the path through sort that notices low memory conditions and causes the sort to spill (out of memory condition management). Also check to make sure we handle queries and fragment failures properly under these conditions. hashjoin, hashagg, and topn use large amounts of memory,and may be unable to complete if their memory needs can't be met; for all others, the idea is that they can complete if they get their reservation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-2878) FragmentExecutor.closeOutResources() is not called if an exception happens in the Foreman before the fragment executor starts running
[
https://issues.apache.org/jira/browse/DRILL-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deneche A. Hakim updated DRILL-2878:
Attachment: DRILL-2878.3.patch.txt
unit test will fail if the query doesn't fail as expected
FragmentExecutor.closeOutResources() is not called if an exception happens in
the Foreman before the fragment executor starts running
-
Key: DRILL-2878
URL: https://issues.apache.org/jira/browse/DRILL-2878
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Reporter: Deneche A. Hakim
Assignee: Jacques Nadeau
Fix For: 1.0.0
Attachments: DRILL-2878.1.patch.txt, DRILL-2878.2.patch.txt,
DRILL-2878.3.patch.txt
When the Foreman sets up the root FragmentExecutor and it needs to wait for
data from the remote fragments, the fragment manager is recorded in the work
bus and the root fragment executor is not run immediately.
If an exception happens in the Foreman while setting up the remote fragments,
the Foreman cancels all fragments and returns a FAILED message to the client.
Because the root fragment executor was not run it will never call it's
closeOutResources() method and it's fragment context will never be closed.
You can easily reproduce this by running the following unit test:
{noformat}
org.apache.drill.exec.server.TestDrillbitResilience#failsWhenSendingFragments
{noformat}
although the test passes successfully because Drill does report the correct
failure to the client, the memory leak is not detected and will show up after
the test finishes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-2757) Verify operators correctly handle low memory conditions and cancellations
[ https://issues.apache.org/jira/browse/DRILL-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated DRILL-2757: Attachment: DRILL-2757.4.patch.txt rebased on top of master Verify operators correctly handle low memory conditions and cancellations - Key: DRILL-2757 URL: https://issues.apache.org/jira/browse/DRILL-2757 Project: Apache Drill Issue Type: Bug Components: Execution - Flow, Execution - Relational Operators Affects Versions: 0.8.0 Reporter: Chris Westin Assignee: Steven Phillips Priority: Blocker Fix For: 1.0.0 Attachments: DRILL-2757.1.patch.txt, DRILL-2757.2.patch.txt, DRILL-2757.3.patch.txt, DRILL-2757.4.patch.txt Check that the path through sort that notices low memory conditions and causes the sort to spill (out of memory condition management). Also check to make sure we handle queries and fragment failures properly under these conditions. hashjoin, hashagg, and topn use large amounts of memory,and may be unable to complete if their memory needs can't be met; for all others, the idea is that they can complete if they get their reservation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2274) Unable to allocate sv2 buffer after repeated attempts : JOIN, Order by used in query
[
https://issues.apache.org/jira/browse/DRILL-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536842#comment-14536842
]
Deneche A. Hakim commented on DRILL-2274:
-
[~rkins] could you try on master again ? thanks
Unable to allocate sv2 buffer after repeated attempts : JOIN, Order by used
in query
Key: DRILL-2274
URL: https://issues.apache.org/jira/browse/DRILL-2274
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Rahul Challapalli
Assignee: Deneche A. Hakim
Fix For: 1.0.0
Attachments: data.json
git.commit.id.abbrev=6676f2d
The below query fails :
{code}
select sub1.uid from `data.json` sub1 inner join `data.json` sub2 on sub1.uid
= sub2.uid order by sub1.uid;
{code}
Error from the logs :
{code}
2015-02-20 00:24:08,431 [2b1981b0-149e-981b-f83f-512c587321d7:frag:1:2] ERROR
o.a.d.e.w.f.AbstractStatusReporter - Error
66dba4ff-644c-4400-ab84-203256dc2600: Failure while running fragment.
java.lang.RuntimeException:
org.apache.drill.exec.memory.OutOfMemoryException: Unable to allocate sv2
buffer after repeated attempts
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:307)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:96)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:97)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:116)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:303)
[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: org.apache.drill.exec.memory.OutOfMemoryException: Unable to
allocate sv2 buffer after repeated attempts
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:516)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:305)
~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
... 16 common frames omitted
{code}
On a different drillbit in the cluster, I found the below message for the
same run
{code}
2015-02-20 00:24:08,435 [BitServer-6] WARN
o.a.d.exec.rpc.control.WorkEventBus - A fragment message arrived but there
was no registered listener for that message: profile {
state: FAILED
error {
error_id: 66dba4ff-644c-4400-ab84-203256dc2600
endpoint {
address: qa-node191.qa.lab
user_port: 31010
