[jira] [Created] (DRILL-3223) Logical plan deserializer for a join node is broken
Piotr Sokólski created DRILL-3223:
-
Summary: Logical plan deserializer for a join node is broken
Key: DRILL-3223
URL: https://issues.apache.org/jira/browse/DRILL-3223
Project: Apache Drill
Issue Type: Bug
Reporter: Piotr Sokólski
Trying to submit a logical query through the web interface or ./bin/submit_plan
fails with an exception
bq. java.lang.IllegalArgumentException: Conflicting property-based creators:
already had [constructor for org.apache.drill.common.logical.data.Join,
annotations: {interface
com.fasterxml.jackson.annotation.JsonCreator=@com.fasterxml.jackson.annotation.JsonCreator()}],
encountered [constructor for org.apache.drill.common.logical.data.Join,
annotations: {interface
com.fasterxml.jackson.annotation.JsonCreator=@com.fasterxml.jackson.annotation.JsonCreator()}]
(Full queried plan and error message:
https://gist.github.com/pyetras/bf625b6697de62284996)
This is most likely due to the JsonCreator annotation being used in two places
in
https://github.com/apache/drill/blob/master/common/src/main/java/org/apache/drill/common/logical/data/Join.java#L52-52
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
known issue? Problem reading JSON
This seems wrong. I can count the records in a JSON table, but select * doesn't work. Is this a known issue? ted:apache-drill-1.0.0$ bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 May 31, 2015 12:14:52 AM org.glassfish.jersey.server.ApplicationHandler initialize INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26... apache drill 1.0.0 got drill? 0: jdbc:drill:zk=local *select count(*) from cp.`sales_fact_1997_collapsed.json` ;* +-+ | EXPR$0 | +-+ | 86837 | +-+ 1 row selected (1.316 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a BigInt type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 3 Fragment 0:0 [Error Id: 8a9ac2c1-9764-42fd-bdeb-ec0b5e408438 on 192.168.1.38:31010] (state=,code=0) 0: jdbc:drill:zk=local *ALTER SYSTEM SET `store.json.read_numbers_as_double` = true;* +---+-+ | ok | summary | +---+-+ | true | store.json.read_numbers_as_double updated. | +---+-+ 1 row selected (0.086 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a VarChar type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 47 Fragment 0:0
Re: known issue? Problem reading JSON
The first error implies at least one column of your JSON data have mixed integer values and float values. The second error seems implies the column has mixed string and float values. You may try with this option : ALTER session set `store.json.all_text_mode` = true; On Sat, May 30, 2015 at 3:16 PM, Ted Dunning ted.dunn...@gmail.com wrote: This seems wrong. I can count the records in a JSON table, but select * doesn't work. Is this a known issue? ted:apache-drill-1.0.0$ bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 May 31, 2015 12:14:52 AM org.glassfish.jersey.server.ApplicationHandler initialize INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26... apache drill 1.0.0 got drill? 0: jdbc:drill:zk=local *select count(*) from cp.`sales_fact_1997_collapsed.json` ;* +-+ | EXPR$0 | +-+ | 86837 | +-+ 1 row selected (1.316 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a BigInt type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 3 Fragment 0:0 [Error Id: 8a9ac2c1-9764-42fd-bdeb-ec0b5e408438 on 192.168.1.38:31010] (state=,code=0) 0: jdbc:drill:zk=local *ALTER SYSTEM SET `store.json.read_numbers_as_double` = true;* +---+-+ | ok | summary | +---+-+ | true | store.json.read_numbers_as_double updated. | +---+-+ 1 row selected (0.086 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a VarChar type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 47 Fragment 0:0
Re: sample UDF project
How about Maven archetype(s) and/or Lazybones templates, with working code examples instead of empty skeletons? Your examples below would make a perfect project template, it works out of the box yet it is easily refactored with an IDE for someone else's project. Sent from my iPhone On May 30, 2015, at 6:36 PM, Ted Dunning ted.dunn...@gmail.com wrote: Hey, I just built the beginnings of a project to contain sample UDF implementations to minimize the startup time of people wanting to build their own. See https://github.com/mapr-demos/simple-drill-functions Ping me if there are improvements to be had. Also, I am curious how we should make this available. I would like for it be outside of the normal drill distribution so that people can see what a stand-alone project should look like. This is related to the documentation which I found to be correct, but less specific than it should be at the end (or beginning) where I think it should say: clone this repo, run these two commands and see something work in drill. Any thoughts here?
[jira] [Created] (DRILL-3224) DrillTestWrapper test-failure message doesn't show actual values
Daniel Barclay (Drill) created DRILL-3224:
-
Summary: DrillTestWrapper test-failure message doesn't show actual
values
Key: DRILL-3224
URL: https://issues.apache.org/jira/browse/DRILL-3224
Project: Apache Drill
Issue Type: Bug
Components: Tools, Build Test
Reporter: Daniel Barclay (Drill)
Assignee: Steven Phillips
For at least one of the test-failure (validation) error messages from
DrillTestWrapper, the message does not include the actual values.
Note how the message below does tell you what was expected (redundant with
what's in the test source code, but still convenient), but doesn't tell you
what the test actually got (not available elsewhere (before/without debugging)).
{noformat}
java.lang.Exception: Did not find expected record in result set:
`CHARACTER_MAXIMUM_LENGTH` : -1, `COLUMN_NAME` : inttype, `NUMERIC_SCALE` : -1,
`DATA_TYPE` : INTEGER, `NUMERIC_PRECISION` : -1,
at
org.apache.drill.DrillTestWrapper.compareResults(DrillTestWrapper.java:541)
at
org.apache.drill.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:295)
at org.apache.drill.DrillTestWrapper.run(DrillTestWrapper.java:119)
at org.apache.drill.TestBuilder.go(TestBuilder.java:125)
at
org.apache.drill.exec.hive.TestInfoSchemaOnHiveStorage.varCharMaxLengthAndDecimalPrecisionInInfoSchema(TestInfoSchemaOnHiveStorage.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.lang.reflect.Method.invoke(Method.java:606)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: known issue? Problem reading JSON
The second error is stating that you have a column that is a string in one row and a double in another. On Sat, May 30, 2015 at 3:16 PM, Ted Dunning ted.dunn...@gmail.com wrote: This seems wrong. I can count the records in a JSON table, but select * doesn't work. Is this a known issue? ted:apache-drill-1.0.0$ bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 May 31, 2015 12:14:52 AM org.glassfish.jersey.server.ApplicationHandler initialize INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26... apache drill 1.0.0 got drill? 0: jdbc:drill:zk=local *select count(*) from cp.`sales_fact_1997_collapsed.json` ;* +-+ | EXPR$0 | +-+ | 86837 | +-+ 1 row selected (1.316 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a BigInt type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 3 Fragment 0:0 [Error Id: 8a9ac2c1-9764-42fd-bdeb-ec0b5e408438 on 192.168.1.38:31010] (state=,code=0) 0: jdbc:drill:zk=local *ALTER SYSTEM SET `store.json.read_numbers_as_double` = true;* +---+-+ | ok | summary | +---+-+ | true | store.json.read_numbers_as_double updated. | +---+-+ 1 row selected (0.086 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a VarChar type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 47 Fragment 0:0
Re: known issue? Problem reading JSON
OK. But this *is* in a data file that we distribute as part of Drill. Perhaps a better error message is warranted? Also, this seems to be a serious limitation that appears only to be fixable using a sledge-hammer. On Sun, May 31, 2015 at 3:31 AM, Jacques Nadeau jacq...@apache.org wrote: The second error is stating that you have a column that is a string in one row and a double in another. On Sat, May 30, 2015 at 3:16 PM, Ted Dunning ted.dunn...@gmail.com wrote: This seems wrong. I can count the records in a JSON table, but select * doesn't work. Is this a known issue? ted:apache-drill-1.0.0$ bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 May 31, 2015 12:14:52 AM org.glassfish.jersey.server.ApplicationHandler initialize INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26... apache drill 1.0.0 got drill? 0: jdbc:drill:zk=local *select count(*) from cp.`sales_fact_1997_collapsed.json` ;* +-+ | EXPR$0 | +-+ | 86837 | +-+ 1 row selected (1.316 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a BigInt type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 3 Fragment 0:0 [Error Id: 8a9ac2c1-9764-42fd-bdeb-ec0b5e408438 on 192.168.1.38:31010] (state=,code=0) 0: jdbc:drill:zk=local *ALTER SYSTEM SET `store.json.read_numbers_as_double` = true;* +---+-+ | ok | summary | +---+-+ | true | store.json.read_numbers_as_double updated. | +---+-+ 1 row selected (0.086 seconds) 0: jdbc:drill:zk=local *select * from cp.`sales_fact_1997_collapsed.json` limit 3;* Error: DATA_READ ERROR: Error parsing JSON - You tried to write a VarChar type when you are using a ValueWriter of type NullableFloat8WriterImpl. File /sales_fact_1997_collapsed.json Record 47 Fragment 0:0
Re: sample UDF project
Empty skeletons? This project has working code. Besides, in my limited survey of development methods, git clone dominates over maven archetypes among developers I know. On Sun, May 31, 2015 at 1:39 AM, Matt Burgess mattyb...@gmail.com wrote: How about Maven archetype(s) and/or Lazybones templates, with working code examples instead of empty skeletons? Your examples below would make a perfect project template, it works out of the box yet it is easily refactored with an IDE for someone else's project. Sent from my iPhone On May 30, 2015, at 6:36 PM, Ted Dunning ted.dunn...@gmail.com wrote: Hey, I just built the beginnings of a project to contain sample UDF implementations to minimize the startup time of people wanting to build their own. See https://github.com/mapr-demos/simple-drill-functions Ping me if there are improvements to be had. Also, I am curious how we should make this available. I would like for it be outside of the normal drill distribution so that people can see what a stand-alone project should look like. This is related to the documentation which I found to be correct, but less specific than it should be at the end (or beginning) where I think it should say: clone this repo, run these two commands and see something work in drill. Any thoughts here?
Review Request 34838: DRILL-3155: Part 1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34838/ --- Review request for drill and Hanifi Gunes. Repository: drill-git Description --- This patch is a simple refactoring. Moved the classes related to complex vectors in the appropriate package. Diffs - exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 7b2b78d exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java 00a78fd exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenTemplate.java b8d040c exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/Flattener.java 323bf43 exec/java-exec/src/main/java/org/apache/drill/exec/store/VectorHolder.java e602fd7 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java 2b929a4 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetRecordReader.java 0cbd480 exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java eddefd0 exec/java-exec/src/main/java/org/apache/drill/exec/vector/BaseRepeatedValueVector.java d5a0d62 exec/java-exec/src/main/java/org/apache/drill/exec/vector/ContainerVectorLike.java 95e3365 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedFixedWidthVectorLike.java 450c673 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedMutator.java 8e097e4 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedValueVector.java 95a7252 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedVariableWidthVectorLike.java ac8589e exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/BaseRepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/ContainerVectorLike.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedFixedWidthVectorLike.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java a5553b2 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMutator.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedVariableWidthVectorLike.java PRE-CREATION Diff: https://reviews.apache.org/r/34838/diff/ Testing --- Thanks, Mehant Baid
Review Request 34839: DRILL-3155: Part 2
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34839/ --- Review request for drill and Hanifi Gunes. Repository: drill-git Description --- While allocating memory for composite vectors if one of the allocation fails we need to release all the allocated memory upto that point. Diffs - exec/java-exec/src/main/codegen/templates/NullableValueVectors.java 90ec6be exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java 7b2b78d exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java b3389e2 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/AbstractMapVector.java 3c01939 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/BaseRepeatedValueVector.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java a97847b Diff: https://reviews.apache.org/r/34839/diff/ Testing --- Thanks, Mehant Baid
[jira] [Created] (DRILL-3222) Need a zip function to combine coordinated lists
Ted Dunning created DRILL-3222: -- Summary: Need a zip function to combine coordinated lists Key: DRILL-3222 URL: https://issues.apache.org/jira/browse/DRILL-3222 Project: Apache Drill Issue Type: Bug Reporter: Ted Dunning It is often very useful to be able to turn a pair (or more) of lists into a single list of pairs. Thus zip([a,b], [1,2]) = [[a,1], [b,2]]. The handling of short lists, more than two lists and so on is TBD, but the base function is an important one. One use case is in time series where storing times as one list and values as another is very handy but processing these results would be much better done by using flatten(zip(times, values)). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
