Re: question about correlated arrays and flatten
That's right. I guess that's what I am proposing to have here implicitly. I am not sure how feasible this would be, however, we should be able to interpret inline lambda like expressions. This is something to discuss as we improve Drill's complex data handling capabilities. I see a great value added here - especially for computationally-intense workloads. select fold(t.numbers, 0, (r, c) = r + c), map(t.numbers, (n) = n*n) from dfs.`some/table` t -Hanifi On Mon, Jun 1, 2015 at 3:28 PM, Ted Dunning ted.dunn...@gmail.com wrote: How could we make functional primitives work without lambda? On Mon, Jun 1, 2015 at 9:55 PM, Hanifi Gunes hgu...@maprtech.com wrote: Idea of having functional primitives with Drill sounds really handy. It would be great if we could support left-right folding as well. I can see many great use cases of project/map, fold/reduce, zip, flatten when combined. On Sat, May 30, 2015 at 12:57 AM, Ted Dunning ted.dunn...@gmail.com wrote: OK. I will file a JIRA for a zip function. No idea if I will be able to get one written in the available cracks of time. On Fri, May 29, 2015 at 7:17 PM, Steven Phillips sphill...@maprtech.com wrote: I think your use case could be solved by adding a UDF that can combine multiple arrays into a single array. The result of this function could then be handled by our current implementation of flatten. I think this is preferable to enhancing flatten itself to handle it, since flatten is not an ordinary UDF, and thus more difficult to modify and maintain. On Fri, May 29, 2015 at 3:20 PM, Ted Dunning ted.dunn...@gmail.com wrote: My particular use case can throw an error if the lists are different length. I think our real goal should be to have a logically complete set of simple primitives that lets any sort of back and forward conversions of this kind. On Fri, May 29, 2015 at 9:58 AM, Jason Altekruse altekruseja...@gmail.com wrote: I understand what you want to do, unfortunately we don't have support for this right now. A UDF is the best I can suggest at this point. Just to explore the idea a little further for the sake of creating a complete feature request, I assume you would just want nulls filled in for the cases where the lists were different lengths? On Fri, May 29, 2015 at 8:58 AM, Ted Dunning ted.dunn...@gmail.com wrote: Input is here: https://gist.github.com/tdunning/07ce66e7e4d4af41afd7 Output is here: https://gist.github.com/tdunning/3aa841c56bfcdc0ab90e log-synth schema for generating input data is here: https://gist.github.com/tdunning/638dd52c00569ffa9582 Preferred syntax would be like select flatten(t, v1, v2) from ... On Fri, May 29, 2015 at 7:04 AM, Neeraja Rentachintala nrentachint...@maprtech.com wrote: Ted can you pls give an example with few data elements in a, b and the expected output you are looking from the query. -Neeraja On Fri, May 29, 2015 at 6:43 AM, Ted Dunning ted.dunn...@gmail.com wrote: I have two arrays. Their elements are correlated times and values. I would like to flatten them into rows, each with two elements. The query select flatten(a), flatten(b) from ... doesn't work because I get the cartesian product (of course). The query select flatten(a, b) from ... also doesn't work because flatten doesn't have a multi-argument form. Going crazy, this query kind of sort of almost works, but not really: select r.x.`key`, flatten(r.x.`value`) from ( select flatten(kvgen(x)) as x from ...) r; What I really want to see is something like this: select zip(flatten(a), flatten(b)) from ... Any pointers? Is my next step to write a UDF? -- Steven Phillips Software Engineer mapr.com
[jira] [Created] (DRILL-3244) Start unit precision getting into fractional seconds precision in View.FieldType serialization
Daniel Barclay (Drill) created DRILL-3244: - Summary: Start unit precision getting into fractional seconds precision in View.FieldType serialization Key: DRILL-3244 URL: https://issues.apache.org/jira/browse/DRILL-3244 Project: Apache Drill Issue Type: Bug Reporter: Daniel Barclay (Drill) For INTERVAL SECOND(n, m), the fractional seconds precision m gets overwritten by the leading unit precision when a View.Field gets serialized and then deserialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34838: DRILL-3155: Part 1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34838/ --- (Updated June 2, 2015, 8:14 p.m.) Review request for drill and Hanifi Gunes. Changes --- Addressed review comments. Repository: drill-git Description --- This patch is a simple refactoring. Moved the classes related to complex vectors in the appropriate package. Diffs (updated) - exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 7b2b78d exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java 00a78fd exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenTemplate.java b8d040c exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/Flattener.java 323bf43 exec/java-exec/src/main/java/org/apache/drill/exec/store/VectorHolder.java e602fd7 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java 2b929a4 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java 0cbd480 exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java eddefd0 exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseRepeatedValueVector.java d5a0d62 exec/java-exec/src/main/java/org/apache/drill/exec/vector/ContainerVectorLike.java 95e3365 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedFixedWidthVectorLike.java 450c673 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedMutator.java 8e097e4 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedValueVector.java 95a7252 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVariableWidthVectorLike.java ac8589e exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/BaseRepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/ContainerVectorLike.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedFixedWidthVectorLike.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java a5553b2 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedVariableWidthVectorLike.java PRE-CREATION Diff: https://reviews.apache.org/r/34838/diff/ Testing --- Thanks, Mehant Baid
HbaseTestSuite failures
Hi Drillers, When I run unit tests (mvn clean install), I am getting: Running org.apache.drill.hbase.HBaseTestsSuite Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 34.951 sec FAILURE! - in org.apache.drill.hbase.HBaseTestsSuite org.apache.drill.hbase.HBaseTestsSuite Time elapsed: 34.951 sec ERROR! java.io.IOException: Shutting down at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:190) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:425) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:93) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:912) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:881) at org.apache.drill.hbase.HBaseTestsSuite.initCluster(HBaseTestsSuite.java:88) Results : Tests in error: HBaseTestsSuite.initCluster:88 » IO Shutting down Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 Is anyone else seeing this? Thank you, Sudheesh
Re: Review Request 34838: DRILL-3155: Part 1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34838/#review86320 --- Ship it! Ship It! - Hanifi Gunes On June 2, 2015, 8:14 p.m., Mehant Baid wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34838/ --- (Updated June 2, 2015, 8:14 p.m.) Review request for drill and Hanifi Gunes. Repository: drill-git Description --- This patch is a simple refactoring. Moved the classes related to complex vectors in the appropriate package. Diffs - exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 7b2b78d exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java 00a78fd exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenTemplate.java b8d040c exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/Flattener.java 323bf43 exec/java-exec/src/main/java/org/apache/drill/exec/store/VectorHolder.java e602fd7 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java 2b929a4 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java 0cbd480 exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java eddefd0 exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseRepeatedValueVector.java d5a0d62 exec/java-exec/src/main/java/org/apache/drill/exec/vector/ContainerVectorLike.java 95e3365 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedFixedWidthVectorLike.java 450c673 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedMutator.java 8e097e4 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedValueVector.java 95a7252 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVariableWidthVectorLike.java ac8589e exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/BaseRepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/ContainerVectorLike.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedFixedWidthVectorLike.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java a5553b2 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedVariableWidthVectorLike.java PRE-CREATION Diff: https://reviews.apache.org/r/34838/diff/ Testing --- Thanks, Mehant Baid
[jira] [Created] (DRILL-3246) Query planning support for partition by clause in Drill's CTAS statement
Jinfeng Ni created DRILL-3246:
-
Summary: Query planning support for partition by clause in Drill's
CTAS statement
Key: DRILL-3246
URL: https://issues.apache.org/jira/browse/DRILL-3246
Project: Apache Drill
Issue Type: New Feature
Components: Query Planning Optimization
Affects Versions: 1.0.0
Reporter: Jinfeng Ni
Assignee: Jinfeng Ni
Fix For: 1.1.0
We are going to add PARTITION BY clause in Drill's CTAS statement. The
PARTITION BY clause will specify the list of columns out of the result
table's column list that will be used to partition the data.
CREATE TABLE table_name [ (col_name, ) ]
[PARTITION BY (col_name, ...)]
AS SELECT_STATEMENT;
Semantics restriction for the PARTITION BY clause:
- All the columns in the PARTITION BY clause have to be in the table's column
list, or the SELECT_STATEMENT has a * column, when the base table in the
SELECT_STATEMENT is schema-less. Otherwise, an query validation error would be
raised.
- When the partition column is resolved to * column in a schema-less query,
this * column could not be a result of join operation. This restriction is
added, since for * out of join operation, query planner would not know which
table might produce this partition column.
Example :
{code}
create table mytable1 partition by (r_regionkey) as
select r_regionkey, r_name from cp.`tpch/region.parquet`
{code}
{code}
create table mytable2 partition by (r_regionkey) as
select * from cp.`tpch/region.parquet`
{code}
{code}
create table mytable3 partition by (r_regionkey) as
select r.r_regionkey, r.r_name, n.n_nationkey, n.n_name
from cp.`tpch/nation.parquet` n, cp.`tpch/region.parquet` r
where n.n_regionkey = r.r_regionkey
{code}
Invalid case 1: Partition column is not in table's column list.
{code}
create table mytable4 partition by (r_regionkey2) as
select r_regionkey, r_name from cp.`tpch/region.parquet`
{code}
Invalid case 2: Partition column is resolved to * out of a join operator.
{code}
create table mytable5 partition by (r_regionkey) as
select *
from cp.`tpch/nation.parquet` n, cp.`tpch/region.parquet` r
where n.n_regionkey = r.r_regionkey
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3248) Explain CTAS statement support
Jinfeng Ni created DRILL-3248: - Summary: Explain CTAS statement support Key: DRILL-3248 URL: https://issues.apache.org/jira/browse/DRILL-3248 Project: Apache Drill Issue Type: New Feature Components: Query Planning Optimization Reporter: Jinfeng Ni Assignee: Jinfeng Ni Currently, only SELECT query could be put in EXPLAIN statement, and Drill does not support EXPLAIN CTAS statement. One option would be to EXPLAIN the select_query in CTAS directly. However, with the partition by clause added in CTAS, it's not possible to do so any more, since there is no corresponding PARTITION BY clause in the SELECT query. Therefore, it is important to extend EXPLAIN to accept CTAS directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34839: DRILL-3155: Part 2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34839/#review86325 --- Ship it! Ship It! - Hanifi Gunes On June 2, 2015, 9:49 p.m., Mehant Baid wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34839/ --- (Updated June 2, 2015, 9:49 p.m.) Review request for drill and Hanifi Gunes. Repository: drill-git Description --- While allocating memory for composite vectors if one of the allocation fails we need to release all the allocated memory upto that point. Diffs - exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 90ec6be exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 7b2b78d exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java b3389e2 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java 3c01939 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/BaseRepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b Diff: https://reviews.apache.org/r/34839/diff/ Testing --- Thanks, Mehant Baid
[jira] [Created] (DRILL-3247) Query without casting results in CompileException
Khurram Faraaz created DRILL-3247:
-
Summary: Query without casting results in CompileException
Key: DRILL-3247
URL: https://issues.apache.org/jira/browse/DRILL-3247
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.0.0
Reporter: Khurram Faraaz
Assignee: Chris Westin
Need a better error message, currently we report CompileException, when columns
are not casted to correct datatypes.
Please see DRILL-3218 for more details.
{code}
0: jdbc:drill:schema=dfs.tmp SELECT MAX(columns[0]) OVER (PARTITION BY
columns[6] ORDER BY columns[4]) FROM `allTypData2.csv`;
Error: SYSTEM ERROR: org.codehaus.commons.compiler.CompileException: Line 330,
Column 31: Unknown variable or type incoming
Fragment 0:0
[Error Id: 285af8f1-ddb4-4d3e-a2d7-bfaef20df5e0 on centos-02.qa.lab:31010]
(state=,code=0)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 34839: DRILL-3155: Part 2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34839/ --- (Updated June 2, 2015, 9:49 p.m.) Review request for drill and Hanifi Gunes. Changes --- Addressed review comments Repository: drill-git Description --- While allocating memory for composite vectors if one of the allocation fails we need to release all the allocated memory upto that point. Diffs (updated) - exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 90ec6be exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 7b2b78d exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java b3389e2 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java 3c01939 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/BaseRepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b Diff: https://reviews.apache.org/r/34839/diff/ Testing --- Thanks, Mehant Baid
Re: HbaseTestSuite failures
This typically indicates that you have another HBase instance running. Make sure you have only one instance running, test run should complete then. Unit tests should not use the same resources with the local HB instance. We should fix this. Did you file a JIRA for this by any chance? -Hanifi 2015-06-02 14:26 GMT-07:00 Abdel Hakim Deneche adene...@maprtech.com: did you try it on master ? On Tue, Jun 2, 2015 at 1:52 PM, Sudheesh Katkam skat...@maprtech.com wrote: Hi Drillers, When I run unit tests (mvn clean install), I am getting: Running org.apache.drill.hbase.HBaseTestsSuite Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 34.951 sec FAILURE! - in org.apache.drill.hbase.HBaseTestsSuite org.apache.drill.hbase.HBaseTestsSuite Time elapsed: 34.951 sec ERROR! java.io.IOException: Shutting down at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:190) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:425) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:93) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:912) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:881) at org.apache.drill.hbase.HBaseTestsSuite.initCluster(HBaseTestsSuite.java:88) Results : Tests in error: HBaseTestsSuite.initCluster:88 » IO Shutting down Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 Is anyone else seeing this? Thank you, Sudheesh -- Abdelhakim Deneche Software Engineer http://www.mapr.com/ Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available
[jira] [Created] (DRILL-3245) Error message needs to be fixed.
Khurram Faraaz created DRILL-3245:
-
Summary: Error message needs to be fixed.
Key: DRILL-3245
URL: https://issues.apache.org/jira/browse/DRILL-3245
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.0.0
Reporter: Khurram Faraaz
Assignee: Chris Westin
The error message need to be fixed.
{code}
0: jdbc:drill:schema=dfs.tmp SELECT SUM(columns[0]) FROM `first_25.csv`;
Error: SYSTEM ERROR: java.lang.RuntimeException: Only COUNT aggregate function
supported for Boolean type
Fragment 0:0
[Error Id: ef5abe03-bbaf-4f20-bf86-4e307f86d944 on centos-02.qa.lab:31010]
(state=,code=0)
{code}
Stack trace from drillbit.log
{code}
[Error Id: eb09fc3d-3b10-4525-b7c4-9e6c66059c3f on centos-02.qa.lab:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
java.lang.RuntimeException: Only COUNT aggregate function supported for Boolean
type
Fragment 0:0
[Error Id: eb09fc3d-3b10-4525-b7c4-9e6c66059c3f on centos-02.qa.lab:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522)
~[drill-common-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:324)
[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:180)
[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:293)
[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.lang.RuntimeException: Only COUNT aggregate function supported
for Boolean type
at
org.apache.drill.exec.test.generated.StreamingAggregatorGen47.setupInterior(StreamingAggTemplate.java:60)
~[na:na]
at
org.apache.drill.exec.test.generated.StreamingAggregatorGen47.setup(StreamingAggTemplate.java:53)
~[na:na]
at
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.createAggregatorInternal(StreamingAggBatch.java:308)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.createAggregator(StreamingAggBatch.java:246)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.buildSchema(StreamingAggBatch.java:113)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:127)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:83)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:80)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:73)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:259)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:253)
~[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.7.0_45]
at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_45]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
~[hadoop-common-2.5.1-mapr-1503.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:253)
[drill-java-exec-1.0.0-mapr-r1-rebuffed.jar:1.0.0-mapr-r1]
... 4 common frames omitted
2015-06-02 21:54:02,686 [BitServer-4] INFO o.a.drill.exec.work.foreman.Foreman
- State change requested. RUNNING -- FAILED
org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
java.lang.RuntimeException: Only COUNT aggregate function supported for Boolean
type
Fragment 0:0
[Error Id: eb09fc3d-3b10-4525-b7c4-9e6c66059c3f on centos-02.qa.lab:31010]
at
org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:458)
Re: HbaseTestSuite failures
did you try it on master ? On Tue, Jun 2, 2015 at 1:52 PM, Sudheesh Katkam skat...@maprtech.com wrote: Hi Drillers, When I run unit tests (mvn clean install), I am getting: Running org.apache.drill.hbase.HBaseTestsSuite Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 34.951 sec FAILURE! - in org.apache.drill.hbase.HBaseTestsSuite org.apache.drill.hbase.HBaseTestsSuite Time elapsed: 34.951 sec ERROR! java.io.IOException: Shutting down at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:190) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:425) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:93) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:912) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:881) at org.apache.drill.hbase.HBaseTestsSuite.initCluster(HBaseTestsSuite.java:88) Results : Tests in error: HBaseTestsSuite.initCluster:88 » IO Shutting down Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 Is anyone else seeing this? Thank you, Sudheesh -- Abdelhakim Deneche Software Engineer http://www.mapr.com/ Now Available - Free Hadoop On-Demand Training http://www.mapr.com/training?utm_source=Emailutm_medium=Signatureutm_campaign=Free%20available
Review Request 34954: DRILL-3196: Disable multiple partition by clauses in the same sql query
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34954/ --- Review request for drill and Aman Sinha. Bugs: DRILL-3196 https://issues.apache.org/jira/browse/DRILL-3196 Repository: drill-git Description --- After validation, detech if there is no than one partition Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java f1ec851 exec/java-exec/src/test/java/org/apache/drill/exec/TestWindowFunctions.java PRE-CREATION Diff: https://reviews.apache.org/r/34954/diff/ Testing --- All the requested tests Thanks, Sean Hsuan-Yi Chu
Review Request 34977: DRILL-3200: Add Window functions: ROW_NUMBER, RANK, PERCENT_RANK, DENSE_RANK and CUME_DIST
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34977/ --- Review request for drill and Steven Phillips. Bugs: DRILL-3200 https://issues.apache.org/jira/browse/DRILL-3200 Repository: drill-git Description --- This is an non final patch, although all required window functions have been implemented the code still needs to be cleaned and properly commented. Here is a list of changes made: - enum WindowFrameRecordBatch.WindowFunction to handle supported window function and their corresponding output MajorType - renamed WindowFrameTemplate - DefaultFrameTemplate, cleaned the template to handle the default frame efficiently: . a batch can be processed as soon as we find the last peer row of it's last row . once a batch is processed it can be safely released = we can transfer it's value vectors to the container instead of copying them - DefaultFrameTemplate.Partition tracks the current window frame and computes the following window functions automatically: row_number, rank, dense_rank, percent_rank, cume_dist. It doesn't need to aggregate the value vectors to compute these window functions - updated TestWindowFrame to check the results of row_number, rank, dense_rank, percent_rank and cume_dist in various cases . added a debug config option to MSorter to control the size of batches. This is needed by TestWindowFrame so it can use small test data files (20 rows per batch) . removed contrib/data/window-test-data - WindowFrameRecordBatch properly releases saved batches if the query stops prematurely - GenerateTestData can be used to generate test data for the window function unit tests [it's a work in progress and can be either improved to make it developer friendly or removed from the final patch] Diffs - contrib/data/pom.xml d1def76 contrib/data/window-test-data/pom.xml 6d195da exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java 91793f5 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/DefaultFrameTemplate.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/Partition.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameRecordBatch.java 428632f exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFrameTemplate.java 78bab54 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/WindowFramer.java 23a2b53 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/MSortTemplate.java 9b21ae3 exec/java-exec/src/main/java/org/apache/drill/exec/record/AbstractRecordBatch.java 330ec79 exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/GenerateTestData.java PRE-CREATION exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java 2b8bd64 exec/java-exec/src/test/resources/window/b1.p1.subs.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b1.p1.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b1.p1/0.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b1.p2.subs.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b1.p2.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b1.p2/0.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b2.p2.subs.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b2.p2.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b2.p2/0.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b2.p2/1.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b2.p4.subs.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b2.p4.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b2.p4/0.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b2.p4/1.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b3.p2.subs.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b3.p2.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b3.p2/0.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b3.p2/1.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b3.p2/2.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b4.p4.subs.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b4.p4.tsv PRE-CREATION exec/java-exec/src/test/resources/window/b4.p4/0.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b4.p4/1.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b4.p4/2.data.json PRE-CREATION exec/java-exec/src/test/resources/window/b4.p4/3.data.json PRE-CREATION exec/java-exec/src/test/resources/window/mediumData.json ad86627 exec/java-exec/src/test/resources/window/oneKeyCount.json
Re: Review Request 34603: DRILL-3167: When a query fails, Foreman should wait for all fragments to finish cleaning up before sending a FAILED state to the client
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34603/#review86237 --- Ship it! Looks good. One change I suggest is changing the user display in the web tier to show FAILED rather than FAILING so people aren't confused. - Jacques Nadeau On May 22, 2015, 5:42 p.m., abdelhakim deneche wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34603/ --- (Updated May 22, 2015, 5:42 p.m.) Review request for drill, Chris Westin, Jacques Nadeau, and Sudheesh Katkam. Bugs: DRILL-3167 https://issues.apache.org/jira/browse/DRILL-3167 Repository: drill-git Description --- - In case of a failure the Foreman will cancel all fragments and move to a FAILING state until all fragments are terminated - QueryManager.cancelExecutingFragments() returns false if no fragment available Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 5d07b49 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java 71b77c6 protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java 92afa4f protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryResult.java 474e330 protocol/src/main/protobuf/UserBitShared.proto 68c8612 Diff: https://reviews.apache.org/r/34603/diff/ Testing --- unit tests are passing Thanks, abdelhakim deneche
[jira] [Resolved] (DRILL-2688) Use of ORDER BY on right side of Union All results in SqlValidatorException
[
https://issues.apache.org/jira/browse/DRILL-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Hsuan-Yi Chu resolved DRILL-2688.
--
Resolution: Invalid
Here is the reference:
http://docs.oracle.com/cd/B28359_01/server.111/b28286/queries004.htm#SQLRF52323
Order-by is not associated with the subquery (union-all 's inputs). Instead, it
is applied at the last. Also, I tried this query in Postgres. Order-by is
applied after union-all.
Use of ORDER BY on right side of Union All results in SqlValidatorException
---
Key: DRILL-2688
URL: https://issues.apache.org/jira/browse/DRILL-2688
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Affects Versions: 0.9.0
Environment: 9d92b8e319f2d46e8659d903d355450e15946533 | DRILL-2580:
Exit early from HashJoinBatch if build side is empty | 26.03.2015 @ 16:13:53
EDT
Reporter: Khurram Faraaz
Assignee: Sean Hsuan-Yi Chu
Priority: Minor
Fix For: 1.2.0
Order by in the right leg of Union All results in SqlValidatorException.
Executing the query individually (the query that is on right side of Union
All) gives expected results and no Exceptions are seen. However, executing
the query as part of the right leg of Union All results in Exception.
Tests were executed on 4 node cluster on CentOS. Both sides of Union All got
input from CSV files.
The query below returns correct results when executed individually. (this is
the same query used on right side of Union All query)
{code}
0: jdbc:drill: select columns[0] from `testWindow.csv` t2 where
t2.columns[0] is not null group by columns[0] order by columns[0];
++
| EXPR$0 |
++
| 10 |
| 100|
| 113|
| 119|
| 2 |
| 50 |
| 55 |
| 57 |
| 61 |
| 67 |
| 89 |
++
11 rows selected (0.265 seconds)
{code}
Note that the Union All query works when there is no order by in the right
leg.
{code}
0: jdbc:drill: select columns[0] from `employee.csv` t1 where
t1.columns[0]=1 union all select columns[0] from `testWindow.csv` t2 where
t2.columns[0] is not null group by columns[0];
++
| EXPR$0 |
++
| 1 |
| 100|
| 10 |
| 2 |
| 50 |
| 55 |
| 67 |
| 113|
| 119|
| 89 |
| 57 |
| 61 |
++
12 rows selected (0.175 seconds)
{code}
In this case we do use an order by on the right side of Union All, which
results in SqlValidatorException. Same query as above, except that we have an
order by clause in the query on the right side of Union All.
{code}
0: jdbc:drill: select columns[0] from `employee.csv` t1 where
t1.columns[0]=1 union all select columns[0] from `testWindow.csv` t2 where
t2.columns[0] is not null group by columns[0] order by columns[0];
Query failed: SqlValidatorException: Column 'columns' not found in any table
Error: exception while executing query: Failure while executing query.
(state=,code=0)
{code}
Stack trace from drillbit.log
{code}
2015-04-04 04:31:22,882 [2ae096e5-4085-de95-5642-ae10d320a8f7:foreman] ERROR
o.a.drill.exec.work.foreman.Foreman - Error
c6a60f71-c959-4ae5-ba8c-23b9c7b6bb7e: SqlValidatorException: Column 'columns'
not found in any table
org.apache.drill.exec.planner.sql.QueryInputException: Failure validating
SQL. org.eigenbase.util.EigenbaseContextException: From line 1, column 178 to
line 1, column 184: Column 'columns' not found in any table
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:147)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:773)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:204)
~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_75]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: net.hydromatic.optiq.tools.ValidationException:
org.eigenbase.util.EigenbaseContextException: From line 1, column 178 to line
1, column 184: Column 'columns' not found in any table
at
net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:176)
~[optiq-core-0.9-drill-r20.jar:na]
at
