[jira] [Resolved] (DRILL-4179) Update UDF documentation now that classpath scanning is more strict
[ https://issues.apache.org/jira/browse/DRILL-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem resolved DRILL-4179. -- Resolution: Fixed > Update UDF documentation now that classpath scanning is more strict > --- > > Key: DRILL-4179 > URL: https://issues.apache.org/jira/browse/DRILL-4179 > Project: Apache Drill > Issue Type: Improvement > Components: Documentation >Reporter: Jason Altekruse >Assignee: Julien Le Dem > > A few issues have come up with users that have UDFs that could be found with > 1.0-1.2, but fail to be loaded with 1.3. There were changes in 1.3 to speed > up finding all UDFs on the classpath made the setup a little more strict. > Some discussions on the topic: > DRILL-4178 > http://search-hadoop.com/m/qRVAXvthcn1xIHUm/+add+your+package+to+drill.classpath.scanning&subj=Re+UDFs+and+1+3 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (DRILL-4557) Make complex writer handle also scalars
Julien Le Dem created DRILL-4557: Summary: Make complex writer handle also scalars Key: DRILL-4557 URL: https://issues.apache.org/jira/browse/DRILL-4557 Project: Apache Drill Issue Type: Improvement Reporter: Julien Le Dem Currently complex writer can be used to write array or map but not scalar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4557) Make complex writer handle also scalars
[ https://issues.apache.org/jira/browse/DRILL-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-4557: - Issue Type: Sub-task (was: Improvement) Parent: DRILL-4538 > Make complex writer handle also scalars > --- > > Key: DRILL-4557 > URL: https://issues.apache.org/jira/browse/DRILL-4557 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Julien Le Dem > > Currently complex writer can be used to write array or map but not scalar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4556) UDF with FieldReader parameter reading union type fails compilation
Julien Le Dem created DRILL-4556: Summary: UDF with FieldReader parameter reading union type fails compilation Key: DRILL-4556 URL: https://issues.apache.org/jira/browse/DRILL-4556 Project: Apache Drill Issue Type: Bug Reporter: Julien Le Dem select foo(a) from mixed where a is a union vector (say mixed is a json file where a is a string or an int) Foo is a UDF that has one param defined as a FieldReader the operator compilation fails as the field is produced as a UnionHolder instead of a FieldReader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4556) UDF with FieldReader parameter reading union type fails compilation
[ https://issues.apache.org/jira/browse/DRILL-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-4556: - Issue Type: Sub-task (was: Bug) Parent: DRILL-4538 > UDF with FieldReader parameter reading union type fails compilation > --- > > Key: DRILL-4556 > URL: https://issues.apache.org/jira/browse/DRILL-4556 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Julien Le Dem > > select foo(a) from mixed > where a is a union vector (say mixed is a json file where a is a string or an > int) > Foo is a UDF that has one param defined as a FieldReader > the operator compilation fails as the field is produced as a UnionHolder > instead of a FieldReader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4555) JsonReader does not support nulls in lists
Julien Le Dem created DRILL-4555:
Summary: JsonReader does not support nulls in lists
Key: DRILL-4555
URL: https://issues.apache.org/jira/browse/DRILL-4555
Project: Apache Drill
Issue Type: Bug
Reporter: Julien Le Dem
{noformat}
case VALUE_NULL:
throw UserException.unsupportedError()
.message("Null values are not supported in lists by default. " +
"Please set `store.json.all_text_mode` to true to read lists
containing nulls. " +
"Be advised that this will treat JSON null values as a string
containing the word 'null'.")
.build(logger);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4047) Select with options
[ https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096789#comment-15096789 ] Julien Le Dem commented on DRILL-4047: -- You can refer to unit tests for more examples: https://github.com/apache/drill/blob/1.4.0/exec/java-exec/src/test/java/org/apache/drill/TestSelectWithOption.java > Select with options > --- > > Key: DRILL-4047 > URL: https://issues.apache.org/jira/browse/DRILL-4047 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Reporter: Julien Le Dem >Assignee: Julien Le Dem > Fix For: 1.4.0 > > > Add a mechanism to pass parameters down to the StoragePlugin when writing a > Select statement. > Some discussion here: > http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E > http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4047) Select with options
[
https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096786#comment-15096786
]
Julien Le Dem commented on DRILL-4047:
--
Here is how it works:
Context:
You can query files in a hadoop file system in the following way:
{noformat}
select columns from dfs.`mytable/file.csv`;
{noformat}
The format plugin is selected based on file extension or header according to
the configuration.
This feature enables providing the format plugin and its config in the SQL
query instead as follows:
{noformat}
select columns from table(dfs.`mytable/file.csv`(type => 'text',
fieldDelimiter => ','))
{noformat}
The table(...) syntax tells the SQL parser that we're passing parameters to the
table.
The type parameter is required and selects the Format plugin as defined by its
Config class (as would be done when deserializing from json)
The other parameters are defined by the fields of the same class. (excluding
"extensions" that does not apply in that case)
For example in the case of Text:
https://github.com/apache/drill/blob/1.4.0/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/TextFormatPlugin.java#L135
{noformat}
@JsonTypeName("text") @JsonInclude(Include.NON_DEFAULT)
public static class TextFormatConfig implements FormatPluginConfig {
public List extensions = ImmutableList.of();
public String lineDelimiter = "\n";
public char fieldDelimiter = '\n';
public char quote = '"';
public char escape = '"';
public char comment = '#';
public boolean skipFirstLine = false;
public boolean extractHeader = false;
{noformat}
> Select with options
> ---
>
> Key: DRILL-4047
> URL: https://issues.apache.org/jira/browse/DRILL-4047
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
>Reporter: Julien Le Dem
>Assignee: Julien Le Dem
> Fix For: 1.4.0
>
>
> Add a mechanism to pass parameters down to the StoragePlugin when writing a
> Select statement.
> Some discussion here:
> http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4245) Add "splittable" or "record_delimitter" property in the JSON format plugin configuration.
Julien Le Dem created DRILL-4245: Summary: Add "splittable" or "record_delimitter" property in the JSON format plugin configuration. Key: DRILL-4245 URL: https://issues.apache.org/jira/browse/DRILL-4245 Project: Apache Drill Issue Type: Improvement Components: Storage - JSON Reporter: Julien Le Dem This will allow splitting JSON files that use a record delimiter. For example one line JSON records separated by \n -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4124) Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs
[ https://issues.apache.org/jira/browse/DRILL-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15054719#comment-15054719 ] Julien Le Dem commented on DRILL-4124: -- Hi [~rkins] in the case where an exception is thrown in a close method we will no longer log a warning. Instead it will be attached to any existing exception or simply thrown and propagated up the stack. It may be difficult to design a test case for this as it will happen only in some failure scenarios. It should at least not change current functional behavior. > Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise > in logs > --- > > Key: DRILL-4124 > URL: https://issues.apache.org/jira/browse/DRILL-4124 > Project: Apache Drill > Issue Type: Improvement >Reporter: Julien Le Dem >Assignee: Julien Le Dem > Fix For: 1.4.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3572) Provide a simple interface to append metadata to files and directories (.drill)
[
https://issues.apache.org/jira/browse/DRILL-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036898#comment-15036898
]
Julien Le Dem edited comment on DRILL-3572 at 12/2/15 11:52 PM:
I created separate sub-tickets for several aspects of the dotdrill file.
Each one of them can be implemented independently assuming they are separate
fields in a {{.drill}} file:
{noformat}
{
version: ...
format: {
...
},
schema: {
...
},
error_handling: {
...
}
}
{noformat}
was (Author: julienledem):
I created separate sub-tickets for several aspects of the dotdrill file.
Each one of them can be implemented independently assuming they are separate
fields in a {{.drill}} file:
{noformat}
{
version: ...
format: {
...
},
schema: {
...
},
error_handling: {
...
}
}
{noformat}
> Provide a simple interface to append metadata to files and directories
> (.drill)
> ---
>
> Key: DRILL-3572
> URL: https://issues.apache.org/jira/browse/DRILL-3572
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Jacques Nadeau
> Fix For: Future
>
>
> We need a way to store small amounts of metadata about a file or a collection
> of files. The current thinking was a way to have a "dot drill file" that
> ascribes metadata to a particular asset.
> Initial example file might be something that includes the following:
> {code}
> {
> // Drill version identifier
> version: "dd1"
>
> // Format Plugin Configuration
> format: {
> type: "httpd",
> format: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
> \"%{Cookie}i\""}
> },
>
> // Traits of underlying data (a.k.a physical properties)
> traits: [ // traits of the underlying data
> {type: "sort_nulls_first", columns: ["request.uri", "client.host"]}
> {type: "unique", columns ["abc"]}
> {type: "unqiue", columns ["xy", "zz"]}
> ],
>
> // Mappings between directory names and exposed columns
> dirs: [
> {skip: true}, // don't include this directory name in the directory path.
> {name: "year", type: "integer"},
> {name: "month", type: "integer"},
> {name: "day", type: "integer"}
> ],
> // whether or not a user can add new columns to the table through insert
> rigid_table: true
>
> }
> {code}
> We also need to support adding more machine-generated/managed data such as
> statistics. That should be done using a separate file from the one that is
> human description.
> A user should be able to ascribe this metadata directly through the file
> system as well as through sql commands such as
> {code}
> ALTER TABLE ADD METADATA ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3572) Provide a simple interface to append metadata to files and directories (.drill)
[
https://issues.apache.org/jira/browse/DRILL-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036898#comment-15036898
]
Julien Le Dem commented on DRILL-3572:
--
I created separate sub-tickets for several aspects of the dotdrill file.
Each one of them can be implemented independently assuming they are separate
fields in a {{.drill}} file:
{noformat}
{
version: ...
format: {
...
},
schema: {
...
},
error_handling: {
...
}
}
{noformat}
> Provide a simple interface to append metadata to files and directories
> (.drill)
> ---
>
> Key: DRILL-3572
> URL: https://issues.apache.org/jira/browse/DRILL-3572
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Jacques Nadeau
> Fix For: Future
>
>
> We need a way to store small amounts of metadata about a file or a collection
> of files. The current thinking was a way to have a "dot drill file" that
> ascribes metadata to a particular asset.
> Initial example file might be something that includes the following:
> {code}
> {
> // Drill version identifier
> version: "dd1"
>
> // Format Plugin Configuration
> format: {
> type: "httpd",
> format: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
> \"%{Cookie}i\""}
> },
>
> // Traits of underlying data (a.k.a physical properties)
> traits: [ // traits of the underlying data
> {type: "sort_nulls_first", columns: ["request.uri", "client.host"]}
> {type: "unique", columns ["abc"]}
> {type: "unqiue", columns ["xy", "zz"]}
> ],
>
> // Mappings between directory names and exposed columns
> dirs: [
> {skip: true}, // don't include this directory name in the directory path.
> {name: "year", type: "integer"},
> {name: "month", type: "integer"},
> {name: "day", type: "integer"}
> ],
> // whether or not a user can add new columns to the table through insert
> rigid_table: true
>
> }
> {code}
> We also need to support adding more machine-generated/managed data such as
> statistics. That should be done using a separate file from the one that is
> human description.
> A user should be able to ascribe this metadata directly through the file
> system as well as through sql commands such as
> {code}
> ALTER TABLE ADD METADATA ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4066) support for format in dot drill file
[ https://issues.apache.org/jira/browse/DRILL-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036891#comment-15036891 ] Julien Le Dem commented on DRILL-4066: -- I'm currently not working on this as I'm focusing on something else for now. I did a little bit of investigation on the feasibility. My take is the DynamicDrillTable can have a compound selection object with different FormatPlugin associations for different subset of the paths to read. That will allow using multiple FormatPlugins in case all the dot drill files do not point to the same one. This is useful when the format has changed over time. > support for format in dot drill file > > > Key: DRILL-4066 > URL: https://issues.apache.org/jira/browse/DRILL-4066 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Julien Le Dem > Fix For: Future > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4066) support for format in dot drill file
[ https://issues.apache.org/jira/browse/DRILL-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-4066: - Assignee: (was: Julien Le Dem) > support for format in dot drill file > > > Key: DRILL-4066 > URL: https://issues.apache.org/jira/browse/DRILL-4066 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Julien Le Dem > Fix For: Future > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3572) Provide a simple interface to append metadata to files and directories (.drill)
[
https://issues.apache.org/jira/browse/DRILL-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem updated DRILL-3572:
-
Assignee: (was: Julien Le Dem)
> Provide a simple interface to append metadata to files and directories
> (.drill)
> ---
>
> Key: DRILL-3572
> URL: https://issues.apache.org/jira/browse/DRILL-3572
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Jacques Nadeau
> Fix For: Future
>
>
> We need a way to store small amounts of metadata about a file or a collection
> of files. The current thinking was a way to have a "dot drill file" that
> ascribes metadata to a particular asset.
> Initial example file might be something that includes the following:
> {code}
> {
> // Drill version identifier
> version: "dd1"
>
> // Format Plugin Configuration
> format: {
> type: "httpd",
> format: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
> \"%{Cookie}i\""}
> },
>
> // Traits of underlying data (a.k.a physical properties)
> traits: [ // traits of the underlying data
> {type: "sort_nulls_first", columns: ["request.uri", "client.host"]}
> {type: "unique", columns ["abc"]}
> {type: "unqiue", columns ["xy", "zz"]}
> ],
>
> // Mappings between directory names and exposed columns
> dirs: [
> {skip: true}, // don't include this directory name in the directory path.
> {name: "year", type: "integer"},
> {name: "month", type: "integer"},
> {name: "day", type: "integer"}
> ],
> // whether or not a user can add new columns to the table through insert
> rigid_table: true
>
> }
> {code}
> We also need to support adding more machine-generated/managed data such as
> statistics. That should be done using a separate file from the one that is
> human description.
> A user should be able to ascribe this metadata directly through the file
> system as well as through sql commands such as
> {code}
> ALTER TABLE ADD METADATA ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-4124) Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs
Julien Le Dem created DRILL-4124: Summary: Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs Key: DRILL-4124 URL: https://issues.apache.org/jira/browse/DRILL-4124 Project: Apache Drill Issue Type: Improvement Reporter: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-4124) Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise in logs
[ https://issues.apache.org/jira/browse/DRILL-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem reassigned DRILL-4124: Assignee: Julien Le Dem > Make all uses of AutoCloseables use addSuppressed exceptions to avoid noise > in logs > --- > > Key: DRILL-4124 > URL: https://issues.apache.org/jira/browse/DRILL-4124 > Project: Apache Drill > Issue Type: Improvement >Reporter: Julien Le Dem >Assignee: Julien Le Dem > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4111) turn tests off in travis as they don't work there
[ https://issues.apache.org/jira/browse/DRILL-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-4111: - Summary: turn tests off in travis as they don't work there (was: Remove travis ci config file) > turn tests off in travis as they don't work there > - > > Key: DRILL-4111 > URL: https://issues.apache.org/jira/browse/DRILL-4111 > Project: Apache Drill > Issue Type: Task >Reporter: Julien Le Dem >Assignee: Julien Le Dem > > Since the travis build always fails, we should just turn it off for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4035) NPE seen on Functional test run using JDK 8
[
https://issues.apache.org/jira/browse/DRILL-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15015067#comment-15015067
]
Julien Le Dem commented on DRILL-4035:
--
Based on the stacktrace above:
{noformat}
Caused by: java.lang.NullPointerException: null
at
hive.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:247)
~[drill-hive-exec-shaded-1.4.0-SNAPSHOT.jar:1.4.0-SNAPSHOT]
{noformat}
It looks like parquet is coming from the drill-hive-exec-shaded jar.
I believe the version of Parquet in there should be excluded.
> NPE seen on Functional test run using JDK 8
> ---
>
> Key: DRILL-4035
> URL: https://issues.apache.org/jira/browse/DRILL-4035
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.3.0
> Environment: 4 node cluster CentOS
>Reporter: Khurram Faraaz
>Assignee: Sudheesh Katkam
> Fix For: 1.4.0
>
>
> I am seeing an NPE in the Functional test run using JDK8 and Drill 1.3
> Failing test is :
> Functional/partition_pruning/hive/parquet/dynamic_hier_intint/data/parquetCount1.q
> select count(*) from
> hive.dynamic_partitions.lineitem_parquet_partitioned_hive_hier_intint;
> {code}
> Drill version was, git.commit.id=e4b94a78
> root@centos drill-1.3.0]# java -version
> openjdk version "1.8.0_65"
> OpenJDK Runtime Environment (build 1.8.0_65-b17)
> OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
> [root@centos drill-1.3.0]# javac -version
> javac 1.8.0_65
> {code}
> {code}
> 2015-11-05 01:37:45 INFO DrillTestJdbc:76 - running test
> /root/public_framework/drill-test-framework/framework/resources/Functional/window_functions/last_val/lastValFn_9.q
> 981260622
> 2015-11-05 01:37:45 INFO DrillResultSetImpl$ResultsListener:1470 - [#137]
> Query failed:
> oadd.org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> NullPointerException
> Fragment 0:0
> [Error Id: cefc7238-a646-4f9a-b4f2-0bd102efe393 on centos-01.qa.lab:31010]
> at
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118)
> at
> oadd.org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:110)
> at
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47)
> at
> oadd.org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32)
> at oadd.org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61)
> at
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233)
> at
> oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205)
> at
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
> at
> oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
> at
> oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
> at
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
> at
> oadd.io.netty.channel.nio.
[jira] [Created] (DRILL-4111) Remove travis ci config file
Julien Le Dem created DRILL-4111: Summary: Remove travis ci config file Key: DRILL-4111 URL: https://issues.apache.org/jira/browse/DRILL-4111 Project: Apache Drill Issue Type: Task Reporter: Julien Le Dem Assignee: Julien Le Dem Since the travis build always fails, we should just turn it off for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-4082) Better error message when multiple versions of the same function are found by the classpath scanner
[ https://issues.apache.org/jira/browse/DRILL-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem resolved DRILL-4082. -- Resolution: Fixed > Better error message when multiple versions of the same function are found by > the classpath scanner > --- > > Key: DRILL-4082 > URL: https://issues.apache.org/jira/browse/DRILL-4082 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Reporter: Julien Le Dem >Assignee: Julien Le Dem > > PR: > https://github.com/apache/drill/pull/252 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4082) Better error message when multiple versions of the same function are found by the classpath scanner
[ https://issues.apache.org/jira/browse/DRILL-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-4082: - Description: PR: https://github.com/apache/drill/pull/252 > Better error message when multiple versions of the same function are found by > the classpath scanner > --- > > Key: DRILL-4082 > URL: https://issues.apache.org/jira/browse/DRILL-4082 > Project: Apache Drill > Issue Type: Bug > Components: Functions - Drill >Reporter: Julien Le Dem >Assignee: Julien Le Dem > > PR: > https://github.com/apache/drill/pull/252 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4082) Better error message when multiple versions of the same function are found by the classpath scanner
Julien Le Dem created DRILL-4082: Summary: Better error message when multiple versions of the same function are found by the classpath scanner Key: DRILL-4082 URL: https://issues.apache.org/jira/browse/DRILL-4082 Project: Apache Drill Issue Type: Bug Components: Functions - Drill Reporter: Julien Le Dem Assignee: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4070) Metadata Caching : min/max values are null for varchar columns in auto partitioned data
[
https://issues.apache.org/jira/browse/DRILL-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15002607#comment-15002607
]
Julien Le Dem commented on DRILL-4070:
--
[~jaltekruse] For future proofing we should probably add Drill version in the
parquet footer. in the extra_metadata field.
> Metadata Caching : min/max values are null for varchar columns in auto
> partitioned data
> ---
>
> Key: DRILL-4070
> URL: https://issues.apache.org/jira/browse/DRILL-4070
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.3.0
>Reporter: Rahul Challapalli
>Priority: Critical
> Attachments: cache.txt, fewtypes_varcharpartition.tar.tgz
>
>
> git.commit.id.abbrev=e78e286
> The metadata cache file created contains incorrect values for min/max fields
> for varchar colums. The data is also partitioned on the varchar column
> {code}
> refresh table metadata fewtypes_varcharpartition;
> {code}
> As a result partition pruning is not happening. This was working after
> DRILL-3937 has been fixed (d331330efd27dbb8922024c4a18c11e76a00016b)
> I attached the data set and the cache file
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3572) Provide a simple interface to append metadata to files and directories (.drill)
[
https://issues.apache.org/jira/browse/DRILL-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem updated DRILL-3572:
-
Issue Type: Improvement (was: Bug)
> Provide a simple interface to append metadata to files and directories
> (.drill)
> ---
>
> Key: DRILL-3572
> URL: https://issues.apache.org/jira/browse/DRILL-3572
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Jacques Nadeau
>Assignee: Julien Le Dem
> Fix For: Future
>
>
> We need a way to store small amounts of metadata about a file or a collection
> of files. The current thinking was a way to have a "dot drill file" that
> ascribes metadata to a particular asset.
> Initial example file might be something that includes the following:
> {code}
> {
> // Drill version identifier
> version: "dd1"
>
> // Format Plugin Configuration
> format: {
> type: "httpd",
> format: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
> \"%{Cookie}i\""}
> },
>
> // Traits of underlying data (a.k.a physical properties)
> traits: [ // traits of the underlying data
> {type: "sort_nulls_first", columns: ["request.uri", "client.host"]}
> {type: "unique", columns ["abc"]}
> {type: "unqiue", columns ["xy", "zz"]}
> ],
>
> // Mappings between directory names and exposed columns
> dirs: [
> {skip: true}, // don't include this directory name in the directory path.
> {name: "year", type: "integer"},
> {name: "month", type: "integer"},
> {name: "day", type: "integer"}
> ],
> // whether or not a user can add new columns to the table through insert
> rigid_table: true
>
> }
> {code}
> We also need to support adding more machine-generated/managed data such as
> statistics. That should be done using a separate file from the one that is
> human description.
> A user should be able to ascribe this metadata directly through the file
> system as well as through sql commands such as
> {code}
> ALTER TABLE ADD METADATA ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (DRILL-4066) support for format in dot drill file
[ https://issues.apache.org/jira/browse/DRILL-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem reassigned DRILL-4066: Assignee: Julien Le Dem > support for format in dot drill file > > > Key: DRILL-4066 > URL: https://issues.apache.org/jira/browse/DRILL-4066 > Project: Apache Drill > Issue Type: Sub-task > Components: Storage - Other >Reporter: Julien Le Dem >Assignee: Julien Le Dem > Fix For: Future > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4068) Error handling in dot drill file
Julien Le Dem created DRILL-4068: Summary: Error handling in dot drill file Key: DRILL-4068 URL: https://issues.apache.org/jira/browse/DRILL-4068 Project: Apache Drill Issue Type: Sub-task Reporter: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4067) support schema configuration in dot drill file
Julien Le Dem created DRILL-4067: Summary: support schema configuration in dot drill file Key: DRILL-4067 URL: https://issues.apache.org/jira/browse/DRILL-4067 Project: Apache Drill Issue Type: Sub-task Reporter: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4066) support for format in dot drill file
Julien Le Dem created DRILL-4066: Summary: support for format in dot drill file Key: DRILL-4066 URL: https://issues.apache.org/jira/browse/DRILL-4066 Project: Apache Drill Issue Type: Sub-task Reporter: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-3572) Provide a simple interface to append metadata to files and directories (.drill)
[
https://issues.apache.org/jira/browse/DRILL-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem reassigned DRILL-3572:
Assignee: Julien Le Dem
> Provide a simple interface to append metadata to files and directories
> (.drill)
> ---
>
> Key: DRILL-3572
> URL: https://issues.apache.org/jira/browse/DRILL-3572
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Other
>Reporter: Jacques Nadeau
>Assignee: Julien Le Dem
> Fix For: Future
>
>
> We need a way to store small amounts of metadata about a file or a collection
> of files. The current thinking was a way to have a "dot drill file" that
> ascribes metadata to a particular asset.
> Initial example file might be something that includes the following:
> {code}
> {
> // Drill version identifier
> version: "dd1"
>
> // Format Plugin Configuration
> format: {
> type: "httpd",
> format: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
> \"%{Cookie}i\""}
> },
>
> // Traits of underlying data (a.k.a physical properties)
> traits: [ // traits of the underlying data
> {type: "sort_nulls_first", columns: ["request.uri", "client.host"]}
> {type: "unique", columns ["abc"]}
> {type: "unqiue", columns ["xy", "zz"]}
> ],
>
> // Mappings between directory names and exposed columns
> dirs: [
> {skip: true}, // don't include this directory name in the directory path.
> {name: "year", type: "integer"},
> {name: "month", type: "integer"},
> {name: "day", type: "integer"}
> ],
> // whether or not a user can add new columns to the table through insert
> rigid_table: true
>
> }
> {code}
> We also need to support adding more machine-generated/managed data such as
> statistics. That should be done using a separate file from the one that is
> human description.
> A user should be able to ascribe this metadata directly through the file
> system as well as through sql commands such as
> {code}
> ALTER TABLE ADD METADATA ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4047) Select with options
[ https://issues.apache.org/jira/browse/DRILL-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-4047: - Description: Add a mechanism to pass parameters down to the StoragePlugin when writing a Select statement. Some discussion here: http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E was: Add a mechanism to pass parameters down to the StoragePlugin when writing a Select statement. Some discussion here: http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E > Select with options > --- > > Key: DRILL-4047 > URL: https://issues.apache.org/jira/browse/DRILL-4047 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Reporter: Julien Le Dem >Assignee: Julien Le Dem > > Add a mechanism to pass parameters down to the StoragePlugin when writing a > Select statement. > Some discussion here: > http://mail-archives.apache.org/mod_mbox/drill-dev/201510.mbox/%3CCAO%2Bvc4AcGK3%2B3QYvQV1-xPPdpG3Tc%2BfG%3D0xDGEUPrhd6ktHv5Q%40mail.gmail.com%3E > http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-4047) Select with options
Julien Le Dem created DRILL-4047: Summary: Select with options Key: DRILL-4047 URL: https://issues.apache.org/jira/browse/DRILL-4047 Project: Apache Drill Issue Type: Improvement Components: Execution - Relational Operators Reporter: Julien Le Dem Assignee: Julien Le Dem Add a mechanism to pass parameters down to the StoragePlugin when writing a Select statement. Some discussion here: http://mail-archives.apache.org/mod_mbox/drill-dev/201511.mbox/%3ccao+vc4clzylvjevisfjqtcyxb-zsmfy4bqrm-jhbidwzgqf...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3994) Build Fails on Windows after DRILL-3742
[
https://issues.apache.org/jira/browse/DRILL-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988048#comment-14988048
]
Julien Le Dem commented on DRILL-3994:
--
[~adityakishore] [~parthc] When running the build on windows I get NPEs when
Hadoop tries to run command lines in o.a.hadoop.util.Shell.runCommand()
Is there something to setup for this to work?
> Build Fails on Windows after DRILL-3742
> ---
>
> Key: DRILL-3994
> URL: https://issues.apache.org/jira/browse/DRILL-3994
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Reporter: Sudheesh Katkam
>Assignee: Julien Le Dem
>Priority: Critical
> Fix For: 1.3.0
>
>
> Build fails on Windows on the latest master:
> {code}
> c:\drill> mvn clean install -DskipTests
> ...
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 169 licence.
> [INFO]
> [INFO] <<< exec-maven-plugin:1.2.1:java (default) < validate @ drill-common
> <<<
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:java (default) @ drill-common ---
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder
> for further details.
> Scanning: C:\drill\common\target\classes
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/]
> at
> org.apache.drill.common.scanner.BuildTimeScan.main(BuildTimeScan.java:129)
> ... 6 more
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Drill Root POM .. SUCCESS [ 10.016
> s]
> [INFO] tools/Parent Pom ... SUCCESS [ 1.062
> s]
> [INFO] tools/freemarker codegen tooling ... SUCCESS [ 6.922
> s]
> [INFO] Drill Protocol . SUCCESS [ 10.062
> s]
> [INFO] Common (Logical Plan, Base expressions) FAILURE [ 9.954
> s]
> [INFO] contrib/Parent Pom . SKIPPED
> [INFO] contrib/data/Parent Pom SKIPPED
> [INFO] contrib/data/tpch-sample-data .. SKIPPED
> [INFO] exec/Parent Pom SKIPPED
> [INFO] exec/Java Execution Engine . SKIPPED
> [INFO] exec/JDBC Driver using dependencies SKIPPED
> [INFO] JDBC JAR with all dependencies . SKIPPED
> [INFO] contrib/mongo-storage-plugin ... SKIPPED
> [INFO] contrib/hbase-storage-plugin ... SKIPPED
> [INFO] contrib/jdbc-storage-plugin SKIPPED
> [INFO] contrib/hive-storage-plugin/Parent Pom . SKIPPED
> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ... SKIPPED
> [INFO] contrib/hive-storage-plugin/core ... SKIPPED
> [INFO] contrib/drill-gis-plugin ... SKIPPED
> [INFO] Packaging and Distribution Assembly SKIPPED
> [INFO] contrib/sqlline SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 38.813 s
> [INFO] Finished at: 2015-10-28T12:17:19-07:00
> [INFO] Final Memory: 67M/466M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java
> (default) on project drill-common: An exception occured while executing the
> Java class. null: InvocationTargetException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/co
[jira] [Commented] (DRILL-3994) Build Fails on Windows after DRILL-3742
[
https://issues.apache.org/jira/browse/DRILL-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988040#comment-14988040
]
Julien Le Dem commented on DRILL-3994:
--
I have fixed the problem and verified it works.
I have run the build on windows and this part works.
I have other test failures that seem unrelated.
> Build Fails on Windows after DRILL-3742
> ---
>
> Key: DRILL-3994
> URL: https://issues.apache.org/jira/browse/DRILL-3994
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Reporter: Sudheesh Katkam
>Assignee: Julien Le Dem
>Priority: Critical
> Fix For: 1.3.0
>
>
> Build fails on Windows on the latest master:
> {code}
> c:\drill> mvn clean install -DskipTests
> ...
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 169 licence.
> [INFO]
> [INFO] <<< exec-maven-plugin:1.2.1:java (default) < validate @ drill-common
> <<<
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:java (default) @ drill-common ---
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder
> for further details.
> Scanning: C:\drill\common\target\classes
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/]
> at
> org.apache.drill.common.scanner.BuildTimeScan.main(BuildTimeScan.java:129)
> ... 6 more
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Drill Root POM .. SUCCESS [ 10.016
> s]
> [INFO] tools/Parent Pom ... SUCCESS [ 1.062
> s]
> [INFO] tools/freemarker codegen tooling ... SUCCESS [ 6.922
> s]
> [INFO] Drill Protocol . SUCCESS [ 10.062
> s]
> [INFO] Common (Logical Plan, Base expressions) FAILURE [ 9.954
> s]
> [INFO] contrib/Parent Pom . SKIPPED
> [INFO] contrib/data/Parent Pom SKIPPED
> [INFO] contrib/data/tpch-sample-data .. SKIPPED
> [INFO] exec/Parent Pom SKIPPED
> [INFO] exec/Java Execution Engine . SKIPPED
> [INFO] exec/JDBC Driver using dependencies SKIPPED
> [INFO] JDBC JAR with all dependencies . SKIPPED
> [INFO] contrib/mongo-storage-plugin ... SKIPPED
> [INFO] contrib/hbase-storage-plugin ... SKIPPED
> [INFO] contrib/jdbc-storage-plugin SKIPPED
> [INFO] contrib/hive-storage-plugin/Parent Pom . SKIPPED
> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ... SKIPPED
> [INFO] contrib/hive-storage-plugin/core ... SKIPPED
> [INFO] contrib/drill-gis-plugin ... SKIPPED
> [INFO] Packaging and Distribution Assembly SKIPPED
> [INFO] contrib/sqlline SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 38.813 s
> [INFO] Finished at: 2015-10-28T12:17:19-07:00
> [INFO] Final Memory: 67M/466M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java
> (default) on project drill-common: An exception occured while executing the
> Java class. null: InvocationTargetException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[jira] [Commented] (DRILL-3972) Vectorize Parquet Writer
[ https://issues.apache.org/jira/browse/DRILL-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987974#comment-14987974 ] Julien Le Dem commented on DRILL-3972: -- yes just create a branch from master and rebase it as needed. Tests should pass. If you see flaky tests or things that do not work out of the box please report them. I think you need to create a subclass of CloseableRecordBatch yes. > Vectorize Parquet Writer > > > Key: DRILL-3972 > URL: https://issues.apache.org/jira/browse/DRILL-3972 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Reporter: Julien Le Dem >Assignee: Andrew Musselman > Labels: pick-me-up > > Currently the > [ParquetRecordWriter|https://github.com/apache/drill/blob/a98da39dd5a8fa368afd8765f4e981826bbfcc0f/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java] > receives one record at a time and then turns that into columns. > Which means we convert from Drill columns to rows and then to Parquet columns. > Instead we could directly convert the Drill columns into Parquet columns in a > vectorized manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3994) Build Fails on Windows after DRILL-3742
[
https://issues.apache.org/jira/browse/DRILL-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983626#comment-14983626
]
Julien Le Dem commented on DRILL-3994:
--
I have not tested it on windows but here a potential fix:
https://github.com/apache/drill/pull/226
> Build Fails on Windows after DRILL-3742
> ---
>
> Key: DRILL-3994
> URL: https://issues.apache.org/jira/browse/DRILL-3994
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Reporter: Sudheesh Katkam
>Assignee: Julien Le Dem
>Priority: Critical
> Fix For: 1.3.0
>
>
> Build fails on Windows on the latest master:
> {code}
> c:\drill> mvn clean install -DskipTests
> ...
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 169 licence.
> [INFO]
> [INFO] <<< exec-maven-plugin:1.2.1:java (default) < validate @ drill-common
> <<<
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:java (default) @ drill-common ---
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder
> for further details.
> Scanning: C:\drill\common\target\classes
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/]
> at
> org.apache.drill.common.scanner.BuildTimeScan.main(BuildTimeScan.java:129)
> ... 6 more
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Drill Root POM .. SUCCESS [ 10.016
> s]
> [INFO] tools/Parent Pom ... SUCCESS [ 1.062
> s]
> [INFO] tools/freemarker codegen tooling ... SUCCESS [ 6.922
> s]
> [INFO] Drill Protocol . SUCCESS [ 10.062
> s]
> [INFO] Common (Logical Plan, Base expressions) FAILURE [ 9.954
> s]
> [INFO] contrib/Parent Pom . SKIPPED
> [INFO] contrib/data/Parent Pom SKIPPED
> [INFO] contrib/data/tpch-sample-data .. SKIPPED
> [INFO] exec/Parent Pom SKIPPED
> [INFO] exec/Java Execution Engine . SKIPPED
> [INFO] exec/JDBC Driver using dependencies SKIPPED
> [INFO] JDBC JAR with all dependencies . SKIPPED
> [INFO] contrib/mongo-storage-plugin ... SKIPPED
> [INFO] contrib/hbase-storage-plugin ... SKIPPED
> [INFO] contrib/jdbc-storage-plugin SKIPPED
> [INFO] contrib/hive-storage-plugin/Parent Pom . SKIPPED
> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ... SKIPPED
> [INFO] contrib/hive-storage-plugin/core ... SKIPPED
> [INFO] contrib/drill-gis-plugin ... SKIPPED
> [INFO] Packaging and Distribution Assembly SKIPPED
> [INFO] contrib/sqlline SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 38.813 s
> [INFO] Finished at: 2015-10-28T12:17:19-07:00
> [INFO] Final Memory: 67M/466M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java
> (default) on project drill-common: An exception occured while executing the
> Java class. null: InvocationTargetException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems,
[jira] [Commented] (DRILL-3994) Build Fails on Windows after DRILL-3742
[
https://issues.apache.org/jira/browse/DRILL-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983615#comment-14983615
]
Julien Le Dem commented on DRILL-3994:
--
I'm looking into it. Currently installing windows.
It looks like the problem is at the following line:
https://github.com/apache/drill/blob/e4b94a78487f844be4fe71c4b9bf88b16c7f42f7/common/src/main/java/org/apache/drill/common/scanner/BuildTimeScan.java#L126
{noformat}
URL url = new URL("file:" + basePath);
Set markedPaths = ClassPathScanner.getMarkedPaths();
if (!markedPaths.contains(url)) {
throw new IllegalArgumentException(url + " not in " + markedPaths);
}
{noformat}
where url is file:C:/ but getMarkedPaths returns a URL of form file:/C:/...
> Build Fails on Windows after DRILL-3742
> ---
>
> Key: DRILL-3994
> URL: https://issues.apache.org/jira/browse/DRILL-3994
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Reporter: Sudheesh Katkam
>Assignee: Julien Le Dem
>Priority: Critical
> Fix For: 1.3.0
>
>
> Build fails on Windows on the latest master:
> {code}
> c:\drill> mvn clean install -DskipTests
> ...
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 169 licence.
> [INFO]
> [INFO] <<< exec-maven-plugin:1.2.1:java (default) < validate @ drill-common
> <<<
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:java (default) @ drill-common ---
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder
> for further details.
> Scanning: C:\drill\common\target\classes
> [WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalArgumentException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/drill/common/target/classes/]
> at
> org.apache.drill.common.scanner.BuildTimeScan.main(BuildTimeScan.java:129)
> ... 6 more
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Drill Root POM .. SUCCESS [ 10.016
> s]
> [INFO] tools/Parent Pom ... SUCCESS [ 1.062
> s]
> [INFO] tools/freemarker codegen tooling ... SUCCESS [ 6.922
> s]
> [INFO] Drill Protocol . SUCCESS [ 10.062
> s]
> [INFO] Common (Logical Plan, Base expressions) FAILURE [ 9.954
> s]
> [INFO] contrib/Parent Pom . SKIPPED
> [INFO] contrib/data/Parent Pom SKIPPED
> [INFO] contrib/data/tpch-sample-data .. SKIPPED
> [INFO] exec/Parent Pom SKIPPED
> [INFO] exec/Java Execution Engine . SKIPPED
> [INFO] exec/JDBC Driver using dependencies SKIPPED
> [INFO] JDBC JAR with all dependencies . SKIPPED
> [INFO] contrib/mongo-storage-plugin ... SKIPPED
> [INFO] contrib/hbase-storage-plugin ... SKIPPED
> [INFO] contrib/jdbc-storage-plugin SKIPPED
> [INFO] contrib/hive-storage-plugin/Parent Pom . SKIPPED
> [INFO] contrib/hive-storage-plugin/hive-exec-shaded ... SKIPPED
> [INFO] contrib/hive-storage-plugin/core ... SKIPPED
> [INFO] contrib/drill-gis-plugin ... SKIPPED
> [INFO] Packaging and Distribution Assembly SKIPPED
> [INFO] contrib/sqlline SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 38.813 s
> [INFO] Finished at: 2015-10-28T12:17:19-07:00
> [INFO] Final Memory: 67M/466M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java
> (default) on project drill-common: An exception occured while executing the
> Java class. null: InvocationTargetException:
> file:C:/drill/common/target/classes/ not in
> [file:/C:/dril
[jira] [Created] (DRILL-3983) Small test improvements
Julien Le Dem created DRILL-3983: Summary: Small test improvements Key: DRILL-3983 URL: https://issues.apache.org/jira/browse/DRILL-3983 Project: Apache Drill Issue Type: Test Reporter: Julien Le Dem Assignee: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3972) Vectorize Parquet Writer
[ https://issues.apache.org/jira/browse/DRILL-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975139#comment-14975139 ] Julien Le Dem commented on DRILL-3972: -- Pull requests sound great. Here the github repo: https://github.com/apache/drill See also: https://drill.apache.org/docs/apache-drill-contribution-guidelines/ > Vectorize Parquet Writer > > > Key: DRILL-3972 > URL: https://issues.apache.org/jira/browse/DRILL-3972 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Reporter: Julien Le Dem > Labels: pick-me-up > > Currently the > [ParquetRecordWriter|https://github.com/apache/drill/blob/a98da39dd5a8fa368afd8765f4e981826bbfcc0f/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java] > receives one record at a time and then turns that into columns. > Which means we convert from Drill columns to rows and then to Parquet columns. > Instead we could directly convert the Drill columns into Parquet columns in a > vectorized manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1721) Configure fmpp-maven-plugin for incremental build
[
https://issues.apache.org/jira/browse/DRILL-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975109#comment-14975109
]
Julien Le Dem commented on DRILL-1721:
--
fixed by:
https://github.com/apache/drill/commit/dbcab0fee8ff416724b770ca08018cf0f17d0035
> Configure fmpp-maven-plugin for incremental build
> -
>
> Key: DRILL-1721
> URL: https://issues.apache.org/jira/browse/DRILL-1721
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 0.7.0
>Reporter: Aditya Kishore
>Assignee: Julien Le Dem
>Priority: Minor
> Fix For: 1.3.0
>
>
> The [FreeMarker Pre Processor maven plugin|
> https://code.google.com/p/freemarkerpp-maven-plugin/], used in {{exec/Java
> Execution Engine}} module recompiles the templates every time, irrespective
> of whether they have changed or not since the last build, generating fresh
> Java sources which further triggers {{javac}} compilation.
> Fixing this will improve the incremental compilation by shaving off close to
> 2 minutes of build time which would be a good win.
> We will have to modify the {{fmpp-maven-plugin}} code. I had attempted this
> in past and a patch may be somewhere in the stash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-1721) Configure fmpp-maven-plugin for incremental build
[
https://issues.apache.org/jira/browse/DRILL-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem resolved DRILL-1721.
--
Resolution: Fixed
Fix Version/s: (was: Future)
1.3.0
> Configure fmpp-maven-plugin for incremental build
> -
>
> Key: DRILL-1721
> URL: https://issues.apache.org/jira/browse/DRILL-1721
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 0.7.0
>Reporter: Aditya Kishore
>Assignee: Julien Le Dem
>Priority: Minor
> Fix For: 1.3.0
>
>
> The [FreeMarker Pre Processor maven plugin|
> https://code.google.com/p/freemarkerpp-maven-plugin/], used in {{exec/Java
> Execution Engine}} module recompiles the templates every time, irrespective
> of whether they have changed or not since the last build, generating fresh
> Java sources which further triggers {{javac}} compilation.
> Fixing this will improve the incremental compilation by shaving off close to
> 2 minutes of build time which would be a good win.
> We will have to modify the {{fmpp-maven-plugin}} code. I had attempted this
> in past and a patch may be somewhere in the stash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (DRILL-1721) Configure fmpp-maven-plugin for incremental build
[
https://issues.apache.org/jira/browse/DRILL-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem reassigned DRILL-1721:
Assignee: Julien Le Dem
> Configure fmpp-maven-plugin for incremental build
> -
>
> Key: DRILL-1721
> URL: https://issues.apache.org/jira/browse/DRILL-1721
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 0.7.0
>Reporter: Aditya Kishore
>Assignee: Julien Le Dem
>Priority: Minor
> Fix For: Future
>
>
> The [FreeMarker Pre Processor maven plugin|
> https://code.google.com/p/freemarkerpp-maven-plugin/], used in {{exec/Java
> Execution Engine}} module recompiles the templates every time, irrespective
> of whether they have changed or not since the last build, generating fresh
> Java sources which further triggers {{javac}} compilation.
> Fixing this will improve the incremental compilation by shaving off close to
> 2 minutes of build time which would be a good win.
> We will have to modify the {{fmpp-maven-plugin}} code. I had attempted this
> in past and a patch may be somewhere in the stash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3972) Vectorize Parquet Writer
[
https://issues.apache.org/jira/browse/DRILL-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974941#comment-14974941
]
Julien Le Dem commented on DRILL-3972:
--
Thanks [~andrew.musselman]
I think you should start by looking at [ParquetWriterBatchCreator
here|https://github.com/apache/drill/blob/b4d47c56b6484eedbf3f44516338da4cdbfbb7b4/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetWriterBatchCreator.java#L35]
It needs to return a CloseableRecordBatch which is the class that gets the
record batches in Columns.
Currently it returns a
[WriterRecordBatch|https://github.com/apache/drill/blob/b4d47c56b6484eedbf3f44516338da4cdbfbb7b4/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java#L158]
which writes records one row at a time to a RecordWriter.
{noformat}return new WriterRecordBatch(writer, incoming, context,
getRecordWriter(context, writer));{noformat}
WriterRecordBatch is the common class to write to a row oriented format.
But in this case we'd want to directly convert the columns and not go through a
row representation in between
> Vectorize Parquet Writer
>
>
> Key: DRILL-3972
> URL: https://issues.apache.org/jira/browse/DRILL-3972
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Reporter: Julien Le Dem
> Labels: pick-me-up
>
> Currently the
> [ParquetRecordWriter|https://github.com/apache/drill/blob/a98da39dd5a8fa368afd8765f4e981826bbfcc0f/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java]
> receives one record at a time and then turns that into columns.
> Which means we convert from Drill columns to rows and then to Parquet columns.
> Instead we could directly convert the Drill columns into Parquet columns in a
> vectorized manner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3972) Vectorize Parquet Writer
[ https://issues.apache.org/jira/browse/DRILL-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971595#comment-14971595 ] Julien Le Dem commented on DRILL-3972: -- I'm not working on this right now but I'm happy to mentor someone if they want to pick it up. > Vectorize Parquet Writer > > > Key: DRILL-3972 > URL: https://issues.apache.org/jira/browse/DRILL-3972 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Reporter: Julien Le Dem > Labels: pick-me-up > > Currently the > [ParquetRecordWriter|https://github.com/apache/drill/blob/a98da39dd5a8fa368afd8765f4e981826bbfcc0f/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java] > receives one record at a time and then turns that into columns. > Which means we convert from Drill columns to rows and then to Parquet columns. > Instead we could directly convert the Drill columns into Parquet columns in a > vectorized manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3972) Vectorize Parquet Writer
Julien Le Dem created DRILL-3972: Summary: Vectorize Parquet Writer Key: DRILL-3972 URL: https://issues.apache.org/jira/browse/DRILL-3972 Project: Apache Drill Issue Type: Bug Components: Storage - Parquet Reporter: Julien Le Dem Currently the [ParquetRecordWriter|https://github.com/apache/drill/blob/a98da39dd5a8fa368afd8765f4e981826bbfcc0f/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java] receives one record at a time and then turns that into columns. Which means we convert from Drill columns to rows and then to Parquet columns. Instead we could directly convert the Drill columns into Parquet columns in a vectorized manner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3934) Mongo tests start a a mongo cluster on hard coded ports
Julien Le Dem created DRILL-3934: Summary: Mongo tests start a a mongo cluster on hard coded ports Key: DRILL-3934 URL: https://issues.apache.org/jira/browse/DRILL-3934 Project: Apache Drill Issue Type: Test Reporter: Julien Le Dem This fails when running multiple instances of the tests on the same machine (in CI for example) https://github.com/apache/drill/blob/27364122ca0a82c6dd5552f292b2711bb2b28cf5/contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestConstants.java#L24 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3836) TestBitRpc uses hard coded port
[
https://issues.apache.org/jira/browse/DRILL-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906998#comment-14906998
]
Julien Le Dem commented on DRILL-3836:
--
change here:
https://github.com/apache/drill/pull/168
> TestBitRpc uses hard coded port
> ---
>
> Key: DRILL-3836
> URL: https://issues.apache.org/jira/browse/DRILL-3836
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Julien Le Dem
>Assignee: Jacques Nadeau
>
> {noformat}
> int port = 1234;
> DataResponseHandler drp = new BitComTestHandler();
> DataServer server = new DataServer(c, workBus, drp);
> port = server.bind(port, false);
> DrillbitEndpoint ep =
> DrillbitEndpoint.newBuilder().setAddress("localhost").setDataPort(port).build();
> {noformat}
> This will collide if the suite is run in parallel on a machine.
> (like say on a jenkins server)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3836) TestBitRpc uses hard coded port
[
https://issues.apache.org/jira/browse/DRILL-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem updated DRILL-3836:
-
Flags: Patch
> TestBitRpc uses hard coded port
> ---
>
> Key: DRILL-3836
> URL: https://issues.apache.org/jira/browse/DRILL-3836
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Julien Le Dem
>Assignee: Jacques Nadeau
>
> {noformat}
> int port = 1234;
> DataResponseHandler drp = new BitComTestHandler();
> DataServer server = new DataServer(c, workBus, drp);
> port = server.bind(port, false);
> DrillbitEndpoint ep =
> DrillbitEndpoint.newBuilder().setAddress("localhost").setDataPort(port).build();
> {noformat}
> This will collide if the suite is run in parallel on a machine.
> (like say on a jenkins server)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3836) TestBitRpc uses hard coded port
[
https://issues.apache.org/jira/browse/DRILL-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem updated DRILL-3836:
-
External issue URL: https://github.com/apache/drill/pull/168
> TestBitRpc uses hard coded port
> ---
>
> Key: DRILL-3836
> URL: https://issues.apache.org/jira/browse/DRILL-3836
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Julien Le Dem
>Assignee: Jacques Nadeau
>
> {noformat}
> int port = 1234;
> DataResponseHandler drp = new BitComTestHandler();
> DataServer server = new DataServer(c, workBus, drp);
> port = server.bind(port, false);
> DrillbitEndpoint ep =
> DrillbitEndpoint.newBuilder().setAddress("localhost").setDataPort(port).build();
> {noformat}
> This will collide if the suite is run in parallel on a machine.
> (like say on a jenkins server)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3836) TestBitRpc uses hard coded port
[
https://issues.apache.org/jira/browse/DRILL-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906985#comment-14906985
]
Julien Le Dem commented on DRILL-3836:
--
thank you!
> TestBitRpc uses hard coded port
> ---
>
> Key: DRILL-3836
> URL: https://issues.apache.org/jira/browse/DRILL-3836
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Julien Le Dem
>Assignee: Jacques Nadeau
>
> {noformat}
> int port = 1234;
> DataResponseHandler drp = new BitComTestHandler();
> DataServer server = new DataServer(c, workBus, drp);
> port = server.bind(port, false);
> DrillbitEndpoint ep =
> DrillbitEndpoint.newBuilder().setAddress("localhost").setDataPort(port).build();
> {noformat}
> This will collide if the suite is run in parallel on a machine.
> (like say on a jenkins server)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3836) TestBitRpc uses hard coded port
[
https://issues.apache.org/jira/browse/DRILL-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem updated DRILL-3836:
-
Assignee: Jacques Nadeau
> TestBitRpc uses hard coded port
> ---
>
> Key: DRILL-3836
> URL: https://issues.apache.org/jira/browse/DRILL-3836
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Julien Le Dem
>Assignee: Jacques Nadeau
>
> {noformat}
> int port = 1234;
> DataResponseHandler drp = new BitComTestHandler();
> DataServer server = new DataServer(c, workBus, drp);
> port = server.bind(port, false);
> DrillbitEndpoint ep =
> DrillbitEndpoint.newBuilder().setAddress("localhost").setDataPort(port).build();
> {noformat}
> This will collide if the suite is run in parallel on a machine.
> (like say on a jenkins server)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3836) TestBitRpc uses hard coded port
Julien Le Dem created DRILL-3836:
Summary: TestBitRpc uses hard coded port
Key: DRILL-3836
URL: https://issues.apache.org/jira/browse/DRILL-3836
Project: Apache Drill
Issue Type: Bug
Reporter: Julien Le Dem
{noformat}
int port = 1234;
DataResponseHandler drp = new BitComTestHandler();
DataServer server = new DataServer(c, workBus, drp);
port = server.bind(port, false);
DrillbitEndpoint ep =
DrillbitEndpoint.newBuilder().setAddress("localhost").setDataPort(port).build();
{noformat}
This will collide if the suite is run in parallel on a machine.
(like say on a jenkins server)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3806) add metadata for untyped null and simple type promotion
Julien Le Dem created DRILL-3806:
Summary: add metadata for untyped null and simple type promotion
Key: DRILL-3806
URL: https://issues.apache.org/jira/browse/DRILL-3806
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types
Reporter: Julien Le Dem
Assignee: Julien Le Dem
Currently when a field has literal null values in JSON the type will be
assigned as BIGINT by default for lack of better type.
```
{
"a": null
}
```
if later on a is assigned with a string value the query will fail with a schema
change error,
The idea is to capture the notion of "untyped null" and implement simple type
promotion from untyped null to the actual type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[ https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3742: - Flags: Patch External issue URL: https://github.com/apache/drill/pull/148/files see pull request: https://github.com/apache/drill/pull/148/files > Improve classpath scanning to reduce the time it takes > -- > > Key: DRILL-3742 > URL: https://issues.apache.org/jira/browse/DRILL-3742 > Project: Apache Drill > Issue Type: Improvement >Reporter: Julien Le Dem >Assignee: Julien Le Dem > > classpath scanning and function registry take a long time (seconds every > time). > We'd want to avoid loading the classes (use bytecode inspection instead) and > have a build time cache to avoid doing the scanning at startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3742) Improve classpath scanning to reduce the time it takes
Julien Le Dem created DRILL-3742: Summary: Improve classpath scanning to reduce the time it takes Key: DRILL-3742 URL: https://issues.apache.org/jira/browse/DRILL-3742 Project: Apache Drill Issue Type: Improvement Reporter: Julien Le Dem Assignee: Julien Le Dem classpath scanning and function registry take a long time (seconds every time). We'd want to avoid loading the classes (use bytecode inspection instead) and have a build time cache to avoid doing the scanning at startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1721) Configure fmpp-maven-plugin for incremental build
[
https://issues.apache.org/jira/browse/DRILL-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717731#comment-14717731
]
Julien Le Dem commented on DRILL-1721:
--
It seems the code of the plugin is extremely simple:
https://code.google.com/p/freemarkerpp-maven-plugin/source/browse/trunk/fmpp-maven-plugin/src/main/java/com/googlecode/fmppmojo/FMPPMojo.java
> Configure fmpp-maven-plugin for incremental build
> -
>
> Key: DRILL-1721
> URL: https://issues.apache.org/jira/browse/DRILL-1721
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Tools, Build & Test
>Affects Versions: 0.7.0
>Reporter: Aditya Kishore
>Assignee: Aditya Kishore
>Priority: Minor
> Fix For: 1.4.0
>
>
> The [FreeMarker Pre Processor maven plugin|
> https://code.google.com/p/freemarkerpp-maven-plugin/], used in {{exec/Java
> Execution Engine}} module recompiles the templates every time, irrespective
> of whether they have changed or not since the last build, generating fresh
> Java sources which further triggers {{javac}} compilation.
> Fixing this will improve the incremental compilation by shaving off close to
> 2 minutes of build time which would be a good win.
> We will have to modify the {{fmpp-maven-plugin}} code. I had attempted this
> in past and a patch may be somewhere in the stash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3717) Move functions to their own module and preprocess them at build time to speedup startup
[ https://issues.apache.org/jira/browse/DRILL-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3717: - Issue Type: Wish (was: Improvement) > Move functions to their own module and preprocess them at build time to > speedup startup > --- > > Key: DRILL-3717 > URL: https://issues.apache.org/jira/browse/DRILL-3717 > Project: Apache Drill > Issue Type: Wish > Components: Execution - Codegen >Reporter: Julien Le Dem >Assignee: Chris Westin > > all the functions included in the drill distirbution are scanned in the > classpath and then parsed with janino at startup. > This slows down startup and unit tests. > If they were in their own module it would be possible to preprocess at build > time and have a much better startup time, including in tests. > The classpath scanning library used (reflections) already has a mechanism to > persist its result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3717) Move functions to their own module and preprocess them at build time to speedup startup
Julien Le Dem created DRILL-3717: Summary: Move functions to their own module and preprocess them at build time to speedup startup Key: DRILL-3717 URL: https://issues.apache.org/jira/browse/DRILL-3717 Project: Apache Drill Issue Type: Improvement Components: Execution - Codegen Reporter: Julien Le Dem Assignee: Chris Westin all the functions included in the drill distirbution are scanned in the classpath and then parsed with janino at startup. This slows down startup and unit tests. If they were in their own module it would be possible to preprocess at build time and have a much better startup time, including in tests. The classpath scanning library used (reflections) already has a mechanism to persist its result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3709) Memory Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712011#comment-14712011
]
Julien Le Dem edited comment on DRILL-3709 at 8/25/15 9:30 PM:
---
Here is something I added to Pig that made it easy to test queries:
http://pig.apache.org/docs/r0.11.1/api/org/apache/pig/builtin/mock/Storage.html
{noformat}
PigServer pigServer = new PigServer(ExecType.LOCAL);
Data data = resetData(pigServer);
data.set("foo",
tuple("a"),
tuple("b"),
tuple("c")
);
pigServer.registerQuery("A = LOAD 'foo' USING mock.Storage();");
pigServer.registerQuery("STORE A INTO 'bar' USING mock.Storage();");
List out = data.get("bar");
assertEquals(tuple("a"), out.get(0));
assertEquals(tuple("b"), out.get(1));
assertEquals(tuple("c"), out.get(2));
{noformat}
was (Author: julienledem):
Here is something I added to Pig that made it easy to test queries:
http://pig.apache.org/docs/r0.11.1/api/org/apache/pig/builtin/mock/Storage.html
{noformat}
PigServer pigServer = new PigServer(ExecType.LOCAL);
Data data = resetData(pigServer);
data.set("foo",
tuple("a"),
tuple("b"),
tuple("c")
);
pigServer.registerQuery("A = LOAD 'foo' USING mock.Storage();");
pigServer.registerQuery("STORE A INTO 'bar' USING mock.Storage();");
List out = data.get("bar");
assertEquals(tuple("a"), out.get(0));
assertEquals(tuple("b"), out.get(1));
assertEquals(tuple("c"), out.get(2));
{noformat}
> Memory Storage Plugin
> -
>
> Key: DRILL-3709
> URL: https://issues.apache.org/jira/browse/DRILL-3709
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Andrew
>Assignee: Jacques Nadeau
> Fix For: 1.3.0
>
>
> Create an in-memory storage plugin for rapid prototyping, testing, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3709) Memory Storage Plugin
[
https://issues.apache.org/jira/browse/DRILL-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712011#comment-14712011
]
Julien Le Dem commented on DRILL-3709:
--
Here is something I added to Pig that made it easy to test queries:
http://pig.apache.org/docs/r0.11.1/api/org/apache/pig/builtin/mock/Storage.html
{noformat}
PigServer pigServer = new PigServer(ExecType.LOCAL);
Data data = resetData(pigServer);
data.set("foo",
tuple("a"),
tuple("b"),
tuple("c")
);
pigServer.registerQuery("A = LOAD 'foo' USING mock.Storage();");
pigServer.registerQuery("STORE A INTO 'bar' USING mock.Storage();");
List out = data.get("bar");
assertEquals(tuple("a"), out.get(0));
assertEquals(tuple("b"), out.get(1));
assertEquals(tuple("c"), out.get(2));
{noformat}
> Memory Storage Plugin
> -
>
> Key: DRILL-3709
> URL: https://issues.apache.org/jira/browse/DRILL-3709
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Reporter: Andrew
>Assignee: Jacques Nadeau
> Fix For: 1.3.0
>
>
> Create an in-memory storage plugin for rapid prototyping, testing, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3708) Make extra resources available on cp when running tests from eclipse
[ https://issues.apache.org/jira/browse/DRILL-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3708: - Attachment: DRILL-3708.1.patch.txt > Make extra resources available on cp when running tests from eclipse > > > Key: DRILL-3708 > URL: https://issues.apache.org/jira/browse/DRILL-3708 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Julien Le Dem >Assignee: Steven Phillips > Attachments: DRILL-3708.1.patch.txt > > > I modified the build so that extra resources (function source, test datasets) > are added to the resource path list instead of directly copied to the build > target. > This way they become known to maven and copied to the jar the standard way in > instead of copied in an opaque manner. > this allows running the tests from eclipse after a mvn eclipse:eclipse. > Otherwise those resources are missing on the cp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3708) Make extra resources available on cp when running tests from eclipse
Julien Le Dem created DRILL-3708: Summary: Make extra resources available on cp when running tests from eclipse Key: DRILL-3708 URL: https://issues.apache.org/jira/browse/DRILL-3708 Project: Apache Drill Issue Type: Bug Components: Tools, Build & Test Reporter: Julien Le Dem Assignee: Steven Phillips I modified the build so that extra resources (function source, test datasets) are added to the resource path list instead of directly copied to the build target. This way they become known to maven and copied to the jar the standard way in instead of copied in an opaque manner. this allows running the tests from eclipse after a mvn eclipse:eclipse. Otherwise those resources are missing on the cp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3703) TopLevelAllocator has a public static mutable field set as a side effect of its Constructor
Julien Le Dem created DRILL-3703:
Summary: TopLevelAllocator has a public static mutable field set
as a side effect of its Constructor
Key: DRILL-3703
URL: https://issues.apache.org/jira/browse/DRILL-3703
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Reporter: Julien Le Dem
Assignee: Chris Westin
https://github.com/apache/drill/blob/c5f1164351fc4b93ba2ef814ad7ab28a80e2/exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java#L44
{noformat} public static long MAXIMUM_DIRECT_MEMORY;{noformat}
https://github.com/apache/drill/blob/c5f1164351fc4b93ba2ef814ad7ab28a80e2/exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java#L65
{noformat}
private TopLevelAllocator(DrillConfig config, long maximumAllocation, boolean
errorOnLeak){
MAXIMUM_DIRECT_MEMORY = maximumAllocation;
...
}
{noformat}
It seems to be used only in MemoryIterator (memory system table):
https://github.com/apache/drill/blob/c5f1164351fc4b93ba2ef814ad7ab28a80e2/exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/MemoryIterator.java#L66
Which should not need a static variable to access this information.
https://github.com/apache/drill/search?utf8=%E2%9C%93&q=MAXIMUM_DIRECT_MEMORY
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3669) fix missing direct dependency
[ https://issues.apache.org/jira/browse/DRILL-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3669: - Assignee: Jason Altekruse (was: Julien Le Dem) > fix missing direct dependency > - > > Key: DRILL-3669 > URL: https://issues.apache.org/jira/browse/DRILL-3669 > Project: Apache Drill > Issue Type: Bug >Reporter: Julien Le Dem >Assignee: Jason Altekruse > Attachments: DRILL-3669.1.patch.txt, DRILL-3669.2.patch.txt > > > This prevents generating a compiling project with mvn eclipse:eclipse > pull request here: > https://github.com/apache/drill/pull/121/files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3669) fix missing direct dependency
[ https://issues.apache.org/jira/browse/DRILL-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3669: - Description: This prevents generating a compiling project with mvn eclipse:eclipse pull request here: https://github.com/apache/drill/pull/121/files > fix missing direct dependency > - > > Key: DRILL-3669 > URL: https://issues.apache.org/jira/browse/DRILL-3669 > Project: Apache Drill > Issue Type: Bug >Reporter: Julien Le Dem >Assignee: Julien Le Dem > Attachments: DRILL-3669.1.patch.txt, DRILL-3669.2.patch.txt > > > This prevents generating a compiling project with mvn eclipse:eclipse > pull request here: > https://github.com/apache/drill/pull/121/files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3669) fix missing direct dependency
[ https://issues.apache.org/jira/browse/DRILL-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3669: - Attachment: DRILL-3669.2.patch.txt > fix missing direct dependency > - > > Key: DRILL-3669 > URL: https://issues.apache.org/jira/browse/DRILL-3669 > Project: Apache Drill > Issue Type: Bug >Reporter: Julien Le Dem >Assignee: Julien Le Dem > Attachments: DRILL-3669.1.patch.txt, DRILL-3669.2.patch.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3669) fix missing direct dependency
[ https://issues.apache.org/jira/browse/DRILL-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated DRILL-3669: - Attachment: DRILL-3669.1.patch.txt > fix missing direct dependency > - > > Key: DRILL-3669 > URL: https://issues.apache.org/jira/browse/DRILL-3669 > Project: Apache Drill > Issue Type: Bug >Reporter: Julien Le Dem >Assignee: Julien Le Dem > Attachments: DRILL-3669.1.patch.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3669) fix missing direct dependency
Julien Le Dem created DRILL-3669: Summary: fix missing direct dependency Key: DRILL-3669 URL: https://issues.apache.org/jira/browse/DRILL-3669 Project: Apache Drill Issue Type: Bug Reporter: Julien Le Dem Assignee: Julien Le Dem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
