[jira] [Created] (DRILL-3932) exception occurs "drill-simple-mask-function" to pass the data of more than 257 characters "IndexOutOfBoundsException"
Shoji Ohara created DRILL-3932:
--
Summary: exception occurs "drill-simple-mask-function" to pass the
data of more than 257 characters "IndexOutOfBoundsException"
Key: DRILL-3932
URL: https://issues.apache.org/jira/browse/DRILL-3932
Project: Apache Drill
Issue Type: Bug
Reporter: Shoji Ohara
Sample of source "drill-simple-mask-function" to pass the data of more than 257
characters "IndexOutOfBoundsException" exception but is a result of the use of,
or Why is that?
{code:title=example}
SELECT
MASK('12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567',
'*' , 255) FROM `dfs`.`test.json`
{code}
{code:title=exception}
Error: SYSTEM ERROR: IndexOutOfBoundsException: index: 0, length: 257
(expected: range(0, 256))
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3933) Error execute select command line sqlline -u -q
Jon created DRILL-3933: -- Summary: Error execute select command line sqlline -u -q Key: DRILL-3933 URL: https://issues.apache.org/jira/browse/DRILL-3933 Project: Apache Drill Issue Type: Bug Affects Versions: 1.1.0 Reporter: Jon I'm newbie with Drill and Jira, so sorry if this is not the correct site. When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from hive.database.table;' " return: Error: PARSE ERROR: Encountered "." at line 1, column 29. Was expecting one of: "FROM" ... "," ... But, if I used UI, in localhost:8047/query, the query works. Drill is connected to Hive with plugin of course. Is this a bug or something? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3933) Error execute select command line sqlline -u -q
[ https://issues.apache.org/jira/browse/DRILL-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon updated DRILL-3933: --- Description: I'm newbie with Drill and Jira, so sorry if this is not the correct site. When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from hive.database.table;' " return: "select anaconda-ks.cfg build.out install.log install.log.syslog ranger_tutorial sandbox.info start_ambari.sh start_hbase.sh start_solr.sh stop_solr.sh from hive.database.table;" Error: PARSE ERROR: Encountered "." at line 1, column 29. Was expecting one of: "FROM" ... "," ... So, to fix this, i should type all columns to do this one work. But, if I used UI, in localhost:8047/query, the query works. Drill is connected to Hive with plugin of course. Is this a bug or something? Bad conf.? was: I'm newbie with Drill and Jira, so sorry if this is not the correct site. When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from hive.database.table;' " return: Error: PARSE ERROR: Encountered "." at line 1, column 29. Was expecting one of: "FROM" ... "," ... But, if I used UI, in localhost:8047/query, the query works. Drill is connected to Hive with plugin of course. Is this a bug or something? > Error execute select command line sqlline -u -q > --- > > Key: DRILL-3933 > URL: https://issues.apache.org/jira/browse/DRILL-3933 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Jon > Labels: bug, newbie > > I'm newbie with Drill and Jira, so sorry if this is not the correct site. > When I query : "sqlline -u 'jdbc:drill:drillbit=localhost' -q 'select * from > hive.database.table;' " return: > "select anaconda-ks.cfg build.out install.log install.log.syslog > ranger_tutorial sandbox.info start_ambari.sh start_hbase.sh start_solr.sh > stop_solr.sh from hive.database.table;" > Error: PARSE ERROR: Encountered "." at line 1, column 29. > Was expecting one of: > "FROM" ... > "," ... > So, to fix this, i should type all columns to do this one work. > But, if I used UI, in localhost:8047/query, the query works. Drill is > connected to Hive with plugin of course. Is this a bug or something? Bad > conf.? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3242) Enhance RPC layer to offload all request work onto a separate thread.
[ https://issues.apache.org/jira/browse/DRILL-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957116#comment-14957116 ] Sudheesh Katkam commented on DRILL-3242: I think this patch is important for higher concurrency. We should not keep a RPC thread busy in any form. Can you rebase on master? (And open a pull request for more comments, since most developers aren't looking at RB anymore?) > Enhance RPC layer to offload all request work onto a separate thread. > - > > Key: DRILL-3242 > URL: https://issues.apache.org/jira/browse/DRILL-3242 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - RPC >Reporter: Chris Westin >Assignee: Jacques Nadeau > Fix For: 1.3.0 > > Attachments: DRILL-3242.patch > > > Right now, the app is responsible for ensuring that very small amounts of > work are done on the RPC thread. In some cases, the app doesn't do this > correctly. Additionally, in high load situations these small amounts of work > become no trivial. As such, we need to make RPC layer protect itself from > slow requests/responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (DRILL-3242) Enhance RPC layer to offload all request work onto a separate thread.
[ https://issues.apache.org/jira/browse/DRILL-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957116#comment-14957116 ] Sudheesh Katkam edited comment on DRILL-3242 at 10/14/15 3:48 PM: -- I think this patch is important for higher concurrency. We should not keep a RPC thread too busy. Can you rebase on master? (And open a pull request for more comments, since most developers aren't looking at RB anymore?) was (Author: sudheeshkatkam): I think this patch is important for higher concurrency. We should not keep a RPC thread busy in any form. Can you rebase on master? (And open a pull request for more comments, since most developers aren't looking at RB anymore?) > Enhance RPC layer to offload all request work onto a separate thread. > - > > Key: DRILL-3242 > URL: https://issues.apache.org/jira/browse/DRILL-3242 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - RPC >Reporter: Chris Westin >Assignee: Jacques Nadeau > Fix For: 1.3.0 > > Attachments: DRILL-3242.patch > > > Right now, the app is responsible for ensuring that very small amounts of > work are done on the RPC thread. In some cases, the app doesn't do this > correctly. Additionally, in high load situations these small amounts of work > become no trivial. As such, we need to make RPC layer protect itself from > slow requests/responses. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957258#comment-14957258
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42020509
--- Diff:
common/src/main/java/org/apache/drill/common/DeferredException.java ---
@@ -47,6 +75,15 @@ public void addException(final Exception exception) {
Preconditions.checkState(!isClosed);
if (this.exception == null) {
+if (exceptionSupplier == null) {
+ this.exception = exception;
+} else {
+ this.exception = exceptionSupplier.get();
+ if (this.exception == null) {
+this.exception = new RuntimeException("Missing root
exception");
+ }
+ this.exception.addSuppressed(exception);
+}
this.exception = exception;
--- End diff --
should we remove this line ?
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957290#comment-14957290
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42021960
--- Diff:
common/src/main/java/org/apache/drill/common/DeferredException.java ---
@@ -123,6 +160,9 @@ public void suppressingClose(final AutoCloseable
autoCloseable) {
autoCloseable.close();
} catch(final Exception e) {
addException(e);
+ } catch(final Error e) {
+logger.warn("caught Error {}", e);
--- End diff --
why log the error here ? we have a JIRA to try and do reduce error logging
to the minimum [DRILL-2692](https://issues.apache.org/jira/browse/DRILL-2692)
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957291#comment-14957291
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42022118
--- Diff:
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java ---
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common;
+
+/**
+ * Provides functionality comparable to Guava's Closeables for
AutoCloseables.
+ */
+public class DrillAutoCloseables {
+ /**
+ * Constructor. Prevents construction for class of static utilities.
+ */
+ private DrillAutoCloseables() {
+ }
+
+ /**
+ * close() an {@see java.lang.AutoCloseable} without throwing a (checked)
+ * {@see java.lang.Exception}. This wraps the close() call with a
+ * try-catch that will rethrow an Exception wrapped with a
+ * {@see java.lang.RuntimeException}, providing a way to call close()
+ * without having to do the try-catch everywhere or propagate the
Exception.
+ *
+ * @param closeable the AutoCloseable to close; may be null
+ * @throws RuntimeException if an Exception occurs; the Exception is
+ * wrapped by the RuntimeException
+ */
+ public static void closeNoChecked(final AutoCloseable autoCloseable) {
--- End diff --
can't we just add this to org.apache.drill.common.AutoCloseables ? it seems
to serve the same purpose
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957359#comment-14957359
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42026020
--- Diff:
common/src/main/java/org/apache/drill/common/config/DrillConfig.java ---
@@ -278,9 +265,9 @@ public String toString() {
return this.root().render();
}
- public static void main(String[] args) throws Exception {
+ public static void main(String[] args) {
--- End diff --
I think it's safe to just remove the main method altogether.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957398#comment-14957398
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42028942
--- Diff:
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java ---
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common;
+
+/**
+ * Provides functionality comparable to Guava's Closeables for
AutoCloseables.
+ */
+public class DrillAutoCloseables {
+ /**
+ * Constructor. Prevents construction for class of static utilities.
+ */
+ private DrillAutoCloseables() {
+ }
+
+ /**
+ * close() an {@see java.lang.AutoCloseable} without throwing a (checked)
+ * {@see java.lang.Exception}. This wraps the close() call with a
+ * try-catch that will rethrow an Exception wrapped with a
+ * {@see java.lang.RuntimeException}, providing a way to call close()
+ * without having to do the try-catch everywhere or propagate the
Exception.
+ *
+ * @param closeable the AutoCloseable to close; may be null
+ * @throws RuntimeException if an Exception occurs; the Exception is
+ * wrapped by the RuntimeException
+ */
+ public static void closeNoChecked(final AutoCloseable autoCloseable) {
--- End diff --
My plan was to merge those two files once I'm done -- and only leave one of
them. They were created in two different sets of work for the same purpose. Can
we do that later?
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957405#comment-14957405
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42029401
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -369,6 +378,10 @@ public void queryCompleted(QueryState state) {
private void fail(Exception ex) {
logger.debug("Submission failed.", ex);
+ final Throwable cause = ex.getCause();
+ if (cause != null) {
+logger.debug("Submission failure cause.", ex.getCause());
--- End diff --
same comment related to
[DRILL-2692](https://issues.apache.org/jira/browse/DRILL-2692)
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957406#comment-14957406
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42029468
--- Diff:
common/src/main/java/org/apache/drill/common/DeferredException.java ---
@@ -47,6 +75,15 @@ public void addException(final Exception exception) {
Preconditions.checkState(!isClosed);
if (this.exception == null) {
+if (exceptionSupplier == null) {
+ this.exception = exception;
+} else {
+ this.exception = exceptionSupplier.get();
+ if (this.exception == null) {
+this.exception = new RuntimeException("Missing root
exception");
+ }
+ this.exception.addSuppressed(exception);
+}
this.exception = exception;
--- End diff --
Yes, that would be a bug. Done.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957411#comment-14957411
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42029661
--- Diff:
common/src/main/java/org/apache/drill/common/DeferredException.java ---
@@ -123,6 +160,9 @@ public void suppressingClose(final AutoCloseable
autoCloseable) {
autoCloseable.close();
} catch(final Exception e) {
addException(e);
+ } catch(final Error e) {
+logger.warn("caught Error {}", e);
--- End diff --
OK, removed.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[ https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957501#comment-14957501 ] ASF GitHub Bot commented on DRILL-3927: --- Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/199#issuecomment-148153599 I'm against having a caught exception for OutOfMemory. This should be a RuntimeException since it can happen anywhere. Why did you choose to use this instead of the Runtime version? > Use OutOfMemoryException in more places > --- > > Key: DRILL-3927 > URL: https://issues.apache.org/jira/browse/DRILL-3927 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Chris Westin >Assignee: Chris Westin > > The new allocator uses OutOfMemoryException in better ways; some additional > exception handling sites need to catch and handle this exception in > preparation for the new allocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957509#comment-14957509
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user jacques-n commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42035940
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java
---
@@ -82,10 +89,32 @@ public Controller getController() {
return controller;
}
+ private void submit(ExecutorService p, final String name, final
Closeable c) {
--- End diff --
Can we move this to AutoCloseables?
We can have the method be:
AutoCloseables.closeQuietlyInParallel(AutoCloseable... closeables) or something
similar?
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957537#comment-14957537
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user tdunning commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42038370
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java
---
@@ -82,10 +89,32 @@ public Controller getController() {
return controller;
}
+ private void submit(ExecutorService p, final String name, final
Closeable c) {
--- End diff --
Hmm... I would worry about the issues that led Guava to remove
closeQuietly. As soon as possible, it would be good to switch to using
try-with-resources instead of closeQuietly idioms.
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957545#comment-14957545
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42038776
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ImplCreator.java
---
@@ -45,6 +45,7 @@
public class ImplCreator {
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ImplCreator.class);
+ private RootExec root = null;
--- End diff --
`root` is never used, right ?
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957548#comment-14957548
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42039233
--- Diff:
common/src/main/java/org/apache/drill/common/config/DrillConfig.java ---
@@ -278,9 +265,9 @@ public String toString() {
return this.root().render();
}
- public static void main(String[] args) throws Exception {
+ public static void main(String[] args) {
--- End diff --
Done.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957556#comment-14957556
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42039460
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java ---
@@ -369,6 +378,10 @@ public void queryCompleted(QueryState state) {
private void fail(Exception ex) {
logger.debug("Submission failed.", ex);
+ final Throwable cause = ex.getCause();
+ if (cause != null) {
+logger.debug("Submission failure cause.", ex.getCause());
--- End diff --
Removed.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[ https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957572#comment-14957572 ] ASF GitHub Bot commented on DRILL-3927: --- Github user cwestin commented on the pull request: https://github.com/apache/drill/pull/199#issuecomment-148167171 I don't understand your comment, Jacques. OutOfMemoryException exists, and is a lot more specific than RuntimeException. Otherwise, what is OutOfMemoryException for? There are no comments in the file, so I'm forced to guess. > Use OutOfMemoryException in more places > --- > > Key: DRILL-3927 > URL: https://issues.apache.org/jira/browse/DRILL-3927 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Chris Westin >Assignee: Chris Westin > > The new allocator uses OutOfMemoryException in better ways; some additional > exception handling sites need to catch and handle this exception in > preparation for the new allocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957573#comment-14957573
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42040416
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ImplCreator.java
---
@@ -45,6 +45,7 @@
public class ImplCreator {
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(ImplCreator.class);
+ private RootExec root = null;
--- End diff --
Apparently not. Removed.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[ https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957670#comment-14957670 ] ASF GitHub Bot commented on DRILL-3927: --- Github user adeneche commented on a diff in the pull request: https://github.com/apache/drill/pull/199#discussion_r42046967 --- Diff: exec/jdbc-all/pom.xml --- @@ -420,7 +420,6 @@ org/apache/drill/exec/server/rest/** org/apache/drill/exec/rpc/data/** org/apache/drill/exec/rpc/control/** - org/apache/drill/exec/work/** --- End diff -- can you check with @dsbos or rahul if the JDBC test tool runs fine with this change ? > Use OutOfMemoryException in more places > --- > > Key: DRILL-3927 > URL: https://issues.apache.org/jira/browse/DRILL-3927 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Chris Westin >Assignee: Chris Westin > > The new allocator uses OutOfMemoryException in better ways; some additional > exception handling sites need to catch and handle this exception in > preparation for the new allocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957673#comment-14957673
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42047219
--- Diff:
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillConnectionImpl.java ---
@@ -369,24 +370,33 @@ AvaticaFactory getFactory() {
return factory;
}
+ private static void closeOrWarn(final AutoCloseable autoCloseable, final
String message, final Logger logger) {
--- End diff --
should we make this public and move this to
`org.apache.drill.common.AutoCloseables` ?
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957677#comment-14957677
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42047454
--- Diff:
common/src/main/java/org/apache/drill/common/DrillAutoCloseables.java ---
@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common;
+
+/**
+ * Provides functionality comparable to Guava's Closeables for
AutoCloseables.
+ */
+public class DrillAutoCloseables {
+ /**
+ * Constructor. Prevents construction for class of static utilities.
+ */
+ private DrillAutoCloseables() {
+ }
+
+ /**
+ * close() an {@see java.lang.AutoCloseable} without throwing a (checked)
+ * {@see java.lang.Exception}. This wraps the close() call with a
+ * try-catch that will rethrow an Exception wrapped with a
+ * {@see java.lang.RuntimeException}, providing a way to call close()
+ * without having to do the try-catch everywhere or propagate the
Exception.
+ *
+ * @param closeable the AutoCloseable to close; may be null
+ * @throws RuntimeException if an Exception occurs; the Exception is
+ * wrapped by the RuntimeException
+ */
+ public static void closeNoChecked(final AutoCloseable autoCloseable) {
--- End diff --
sounds good to me
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3930) Remove direct references to TopLevelAllocator from unit tests
[ https://issues.apache.org/jira/browse/DRILL-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957696#comment-14957696 ] ASF GitHub Bot commented on DRILL-3930: --- Github user hnfgns commented on the pull request: https://github.com/apache/drill/pull/200#issuecomment-148187351 +1 > Remove direct references to TopLevelAllocator from unit tests > - > > Key: DRILL-3930 > URL: https://issues.apache.org/jira/browse/DRILL-3930 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.2.0 >Reporter: Chris Westin >Assignee: Chris Westin > > The RootAllocatorFactory should be used throughout the code to allow us to > change allocators via configuration or other software choices. Some unit > tests still reference TopLevelAllocator directly. We also need to do a better > job of handling exceptions that can be handled by close()ing an allocator > that isn't in the proper state (remaining open child allocators, outstanding > buffers, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[ https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957722#comment-14957722 ] ASF GitHub Bot commented on DRILL-3927: --- Github user cwestin commented on a diff in the pull request: https://github.com/apache/drill/pull/199#discussion_r42050587 --- Diff: exec/jdbc-all/pom.xml --- @@ -420,7 +420,6 @@ org/apache/drill/exec/server/rest/** org/apache/drill/exec/rpc/data/** org/apache/drill/exec/rpc/control/** - org/apache/drill/exec/work/** --- End diff -- I've sent them email, and copied you, Hakim. > Use OutOfMemoryException in more places > --- > > Key: DRILL-3927 > URL: https://issues.apache.org/jira/browse/DRILL-3927 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Chris Westin >Assignee: Chris Westin > > The new allocator uses OutOfMemoryException in better ways; some additional > exception handling sites need to catch and handle this exception in > preparation for the new allocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[
https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957724#comment-14957724
]
ASF GitHub Bot commented on DRILL-3927:
---
Github user cwestin commented on a diff in the pull request:
https://github.com/apache/drill/pull/199#discussion_r42050675
--- Diff:
exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillConnectionImpl.java ---
@@ -369,24 +370,33 @@ AvaticaFactory getFactory() {
return factory;
}
+ private static void closeOrWarn(final AutoCloseable autoCloseable, final
String message, final Logger logger) {
--- End diff --
Sounds good, but let's leave that until DrillAutoCloseables and
AutoCloseables are merged, along with the change you suggested earlier/above.
> Use OutOfMemoryException in more places
> ---
>
> Key: DRILL-3927
> URL: https://issues.apache.org/jira/browse/DRILL-3927
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Reporter: Chris Westin
>Assignee: Chris Westin
>
> The new allocator uses OutOfMemoryException in better ways; some additional
> exception handling sites need to catch and handle this exception in
> preparation for the new allocator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957734#comment-14957734
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42051351
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java
---
@@ -82,10 +89,32 @@ public Controller getController() {
return controller;
}
+ private void submit(ExecutorService p, final String name, final
Closeable c) {
--- End diff --
I agree about removing closeQuietly but this sounds like a separate PR.
This one is already bigger than it should, so I'm trying not to do too much
refactoring in it.
I'm happy to do the closeQuietly refactoring as a follow up.
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957769#comment-14957769 ] Andrew commented on DRILL-3749: --- So far the tests pass on Hadoop 2.7.1. The only changes I needed to make were in the impersonation tests. Only the error messages were slightly different. > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3911) Upgrade Hadoop from 2.4.1 to latest stable
[ https://issues.apache.org/jira/browse/DRILL-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957807#comment-14957807 ] ASF GitHub Bot commented on DRILL-3911: --- GitHub user aleph-zero opened a pull request: https://github.com/apache/drill/pull/202 DRILL-3911: Upgrade Hadoop version This commit upgrades the Hadoop version from 2.4.1 to 2.7.1. The only change that needed to be made here was to modify the error message that we expect to get back for failed impersonation requests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aleph-zero/drill issues/DRILL-3749 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/202.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #202 commit aace015f731358b89ec15851f71742bd8db2b5e0 Author: aleph-zero Date: 2015-10-08T23:59:48Z DRILL-3911: Upgrade Hadoop version This commit upgrades the Hadoop version from 2.4.1 to 2.7.1. The only change that needed to be made here was to modify the error message that we expect to get back for failed impersonation requests. > Upgrade Hadoop from 2.4.1 to latest stable > -- > > Key: DRILL-3911 > URL: https://issues.apache.org/jira/browse/DRILL-3911 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Reporter: Andrew >Assignee: Andrew > Fix For: 1.3.0 > > > Later versions of Hadoop have improved S3 compatibility > (https://issues.apache.org/jira/browse/HADOOP-10400). > Since users are increasingly using Drill with S3, we should upgrade our > Hadoop dependency so we can get the best integration possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew reassigned DRILL-3749: - Assignee: Andrew > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Andrew > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957809#comment-14957809 ] Andrew commented on DRILL-3749: --- PR: https://github.com/apache/drill/pull/202 > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Andrew > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3911) Upgrade Hadoop from 2.4.1 to latest stable
[ https://issues.apache.org/jira/browse/DRILL-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957825#comment-14957825 ] ASF GitHub Bot commented on DRILL-3911: --- Github user aleph-zero closed the pull request at: https://github.com/apache/drill/pull/202 > Upgrade Hadoop from 2.4.1 to latest stable > -- > > Key: DRILL-3911 > URL: https://issues.apache.org/jira/browse/DRILL-3911 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Other >Reporter: Andrew >Assignee: Andrew > Fix For: 1.3.0 > > > Later versions of Hadoop have improved S3 compatibility > (https://issues.apache.org/jira/browse/HADOOP-10400). > Since users are increasingly using Drill with S3, we should upgrade our > Hadoop dependency so we can get the best integration possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew updated DRILL-3749: -- Comment: was deleted (was: PR: https://github.com/apache/drill/pull/202) > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Andrew > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957843#comment-14957843 ] ASF GitHub Bot commented on DRILL-3749: --- GitHub user aleph-zero opened a pull request: https://github.com/apache/drill/pull/203 DRILL-3749: Upgrade Hadoop to 2.7.1 Upgrades Hadoop dependency from 2.4.1 to 2.7.1. This allows us to take advantage of the latest S3 filesystem support. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aleph-zero/drill issues/DRILL-3749 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/203.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #203 commit a062eba29099f2efcd0e9e41e79b7a5e3485991a Author: aleph-zero Date: 2015-10-14T21:31:57Z DRILL-3749: Upgrade Hadoop to 2.7.1 Upgrades Hadoop dependency from 2.4.1 to 2.7.1. This allows us to take advantage of the latest S3 filesystem support. > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Andrew > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew reassigned DRILL-3749: - Assignee: Jason Altekruse (was: Andrew) > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Jason Altekruse > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Phillips updated DRILL-3749: --- Assignee: Jason Altekruse (was: Steven Phillips) > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Jason Altekruse > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew reassigned DRILL-3749: - Assignee: Steven Phillips (was: Jason Altekruse) > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Steven Phillips > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957857#comment-14957857
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42057395
--- Diff:
common/src/main/java/org/apache/drill/common/logical/StoragePluginConfigBase.java
---
@@ -17,30 +17,18 @@
*/
package org.apache.drill.common.logical;
-import java.util.List;
+import java.util.Set;
-import org.apache.drill.common.config.CommonConstants;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.common.util.PathScanner;
-
-import com.google.common.base.Joiner;
+import org.apache.drill.common.scanner.persistence.ScanResult;
public abstract class StoragePluginConfigBase extends StoragePluginConfig {
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(StoragePluginConfigBase.class);
-
- public synchronized static Class[] getSubTypes(final DrillConfig
config) {
-final List packages =
-
config.getStringList(CommonConstants.STORAGE_PLUGIN_CONFIG_SCAN_PACKAGES);
-final Class[] pluginClasses =
-PathScanner.scanForImplementationsArr(StoragePluginConfig.class,
packages);
-final String lineBrokenList =
-pluginClasses.length == 0
-? "" : "\n\t- " + Joiner.on("\n\t- ").join(pluginClasses);
-logger.debug("Found {} storage plugin configuration classes: {}.",
- pluginClasses.length, lineBrokenList);
-return pluginClasses;
+ public synchronized static Set>
getSubTypes(final ScanResult classpathScan) {
--- End diff --
removed synchronized
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3749) Upgrade Hadoop dependency to latest version (2.7.1)
[ https://issues.apache.org/jira/browse/DRILL-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957858#comment-14957858 ] ASF GitHub Bot commented on DRILL-3749: --- Github user vkorukanti commented on the pull request: https://github.com/apache/drill/pull/203#issuecomment-148210343 Can you check if we need to update (override new methods) the DrillFileSystem and/or DrillFsDataInputStream which are decorators around hadoop FileSystem and FsDataInputStream? > Upgrade Hadoop dependency to latest version (2.7.1) > --- > > Key: DRILL-3749 > URL: https://issues.apache.org/jira/browse/DRILL-3749 > Project: Apache Drill > Issue Type: New Feature > Components: Tools, Build & Test >Affects Versions: 1.1.0 >Reporter: Venki Korukanti >Assignee: Jason Altekruse > Fix For: Future > > > Logging a JIRA to track and discuss upgrading Drill's Hadoop dependency > version. Currently Drill depends on Hadoop 2.5.0 version. Newer version of > Hadoop (2.7.1) has following features. > 1) Better S3 support > 2) Ability to check if a user has certain permissions on file/directory > without performing operations on the file/dir. Useful for cases like > DRILL-3467. > As Drill is going to use higher version of Hadoop fileclient, there could be > potential issues when interacting with Hadoop services (such as HDFS) of > lower version than the fileclient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957867#comment-14957867
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42057945
--- Diff:
common/src/main/java/org/apache/drill/common/scanner/BuildTimeScan.java ---
@@ -0,0 +1,131 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common.scanner;
+
+import static
com.fasterxml.jackson.databind.SerializationFeature.INDENT_OUTPUT;
+import static java.lang.String.format;
+import static java.util.Arrays.asList;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URL;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.DrillRuntimeException;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+import com.fasterxml.jackson.databind.ObjectWriter;
+
+public class BuildTimeScan {
--- End diff --
ok
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957871#comment-14957871
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42058233
--- Diff:
common/src/main/java/org/apache/drill/common/scanner/ClassPathScanner.java ---
@@ -0,0 +1,458 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.common.scanner;
+
+import static java.lang.String.format;
+import static java.util.Arrays.asList;
+import static java.util.Collections.unmodifiableList;
+import static java.util.Collections.unmodifiableSet;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.drill.common.config.CommonConstants;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.scanner.persistence.AnnotationDescriptor;
+import org.apache.drill.common.scanner.persistence.AttributeDescriptor;
+import org.apache.drill.common.scanner.persistence.ChildClassDescriptor;
+import org.apache.drill.common.scanner.persistence.FieldDescriptor;
+import org.apache.drill.common.scanner.persistence.FunctionDescriptor;
+import org.apache.drill.common.scanner.persistence.ParentClassDescriptor;
+import org.apache.drill.common.scanner.persistence.ScanResult;
+import org.reflections.Reflections;
+import org.reflections.adapters.JavassistAdapter;
+import org.reflections.scanners.AbstractScanner;
+import org.reflections.util.ConfigurationBuilder;
+import org.reflections.util.FilterBuilder;
+
+import com.google.common.base.Stopwatch;
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+
+import javassist.bytecode.AccessFlag;
+import javassist.bytecode.AnnotationsAttribute;
+import javassist.bytecode.ClassFile;
+import javassist.bytecode.FieldInfo;
+import javassist.bytecode.annotation.AnnotationMemberValue;
+import javassist.bytecode.annotation.ArrayMemberValue;
+import javassist.bytecode.annotation.BooleanMemberValue;
+import javassist.bytecode.annotation.ByteMemberValue;
+import javassist.bytecode.annotation.CharMemberValue;
+import javassist.bytecode.annotation.ClassMemberValue;
+import javassist.bytecode.annotation.DoubleMemberValue;
+import javassist.bytecode.annotation.EnumMemberValue;
+import javassist.bytecode.annotation.FloatMemberValue;
+import javassist.bytecode.annotation.IntegerMemberValue;
+import javassist.bytecode.annotation.LongMemberValue;
+import javassist.bytecode.annotation.MemberValue;
+import javassist.bytecode.annotation.MemberValueVisitor;
+import javassist.bytecode.annotation.ShortMemberValue;
+import javassist.bytecode.annotation.StringMemberValue;
+
+/**
+ * Classpath scanning utility.
+ * The classpath should be scanned once at startup from a DrillConfig
instance. {@see ClassPathScanner#fromPrescan(DrillConfig)}
+ * The DrillConfig provides the list of packages to scan.
(drill.classpath.scanning.packages) {@see
CommonConstants#IMPLEMENTATIONS_SCAN_PACKAGES}
+ * Only the class directories and jars containing a drill-module.conf will
be scanned.
+ * Drill core packages are scanned at build time and the result is saved
in a JSON file. {@see ClassPathScanner#FUNCTION_REGISTRY_FILE}
+ * At runtime only the packages that have not be scanned yet will be
scanned.
--- End diff --
thanks
> Improve classpath scanning to reduce the time it takes
>
[jira] [Created] (DRILL-3934) Mongo tests start a a mongo cluster on hard coded ports
Julien Le Dem created DRILL-3934: Summary: Mongo tests start a a mongo cluster on hard coded ports Key: DRILL-3934 URL: https://issues.apache.org/jira/browse/DRILL-3934 Project: Apache Drill Issue Type: Test Reporter: Julien Le Dem This fails when running multiple instances of the tests on the same machine (in CI for example) https://github.com/apache/drill/blob/27364122ca0a82c6dd5552f292b2711bb2b28cf5/contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestConstants.java#L24 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957914#comment-14957914
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42060175
--- Diff:
contrib/storage-mongo/src/test/java/org/apache/drill/exec/store/mongo/MongoTestConstants.java
---
@@ -20,6 +20,7 @@
public interface MongoTestConstants {
public static final String LOCALHOST = "localhost";
+ // TODO: add some randomization to this as it fails when running
concurrent builds
--- End diff --
https://issues.apache.org/jira/browse/DRILL-3934
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3935) Modify VarCharWriter to allow passing byte[]
Andrew created DRILL-3935: - Summary: Modify VarCharWriter to allow passing byte[] Key: DRILL-3935 URL: https://issues.apache.org/jira/browse/DRILL-3935 Project: Apache Drill Issue Type: Bug Components: Execution - Data Types Reporter: Andrew Assignee: Andrew Priority: Minor Fix For: 1.3.0 In RecordReader implementations, one sometimes has variable-length string data in the form of byte arrays. It is a cumbersome idiom to pack these into a DrillBuf, only to pass it on to the write. I propose adding this method to the VarCharWriter interface: public void writeVarChar(int start, int end, byte[] bytes); Without this I currently have to do: byte[] bytes = value.getBytes(Charsets.UTF_8); buffer.reallocIfNeeded(bytes.length); buffer.setBytes(0, bytes); writer.varChar().writeVarChar(0, bytes.length, buffer); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[ https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957923#comment-14957923 ] ASF GitHub Bot commented on DRILL-3742: --- Github user julienledem commented on a diff in the pull request: https://github.com/apache/drill/pull/148#discussion_r42060557 --- Diff: exec/java-exec/pom.xml --- @@ -609,6 +559,59 @@ + +org.codehaus.mojo +build-helper-maven-plugin +1.9.1 + + +add-sources-as-resources +process-sources + + add-resource + + + + + src/main/java + + + src/test/java --- End diff -- moving those to test-resources instead > Improve classpath scanning to reduce the time it takes > -- > > Key: DRILL-3742 > URL: https://issues.apache.org/jira/browse/DRILL-3742 > Project: Apache Drill > Issue Type: Improvement >Reporter: Julien Le Dem > Fix For: Future > > > classpath scanning and function registry take a long time (seconds every > time). > We'd want to avoid loading the classes (use bytecode inspection instead) and > have a build time cache to avoid doing the scanning at startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957924#comment-14957924
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42060644
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionAttributes.java
---
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import
org.apache.drill.exec.expr.annotations.FunctionTemplate.FunctionCostCategory;
+import
org.apache.drill.exec.expr.annotations.FunctionTemplate.FunctionScope;
+import
org.apache.drill.exec.expr.annotations.FunctionTemplate.NullHandling;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder.ValueReference;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder.WorkspaceReference;
+
+public class FunctionAttributes {
--- End diff --
will do
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957929#comment-14957929
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42061281
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionInitializer.java
---
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.fn;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.StringReader;
+import java.net.URL;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.util.FileUtils;
+import org.codehaus.commons.compiler.CompileException;
+import org.codehaus.janino.Java.CompilationUnit;
+import org.codehaus.janino.Parser;
+import org.codehaus.janino.Scanner;
+import org.mortbay.util.IO;
+
+import com.google.common.collect.Maps;
+import com.google.common.io.InputSupplier;
+import com.google.common.io.Resources;
+
+public class FunctionInitializer {
--- End diff --
updated
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[ https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957932#comment-14957932 ] ASF GitHub Bot commented on DRILL-3927: --- Github user cwestin commented on a diff in the pull request: https://github.com/apache/drill/pull/199#discussion_r42061711 --- Diff: exec/jdbc-all/pom.xml --- @@ -420,7 +420,6 @@ org/apache/drill/exec/server/rest/** org/apache/drill/exec/rpc/data/** org/apache/drill/exec/rpc/control/** - org/apache/drill/exec/work/** --- End diff -- Daniel checked it out and said it was fine. > Use OutOfMemoryException in more places > --- > > Key: DRILL-3927 > URL: https://issues.apache.org/jira/browse/DRILL-3927 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Chris Westin >Assignee: Chris Westin > > The new allocator uses OutOfMemoryException in better ways; some additional > exception handling sites need to catch and handle this exception in > preparation for the new allocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[ https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957934#comment-14957934 ] ASF GitHub Bot commented on DRILL-3742: --- Github user julienledem commented on a diff in the pull request: https://github.com/apache/drill/pull/148#discussion_r42061773 --- Diff: fmpp/pom.xml --- @@ -0,0 +1,58 @@ + --- End diff -- This is a submodule. tools is not right now, do you want to make tools part of the project module hierarchy? > Improve classpath scanning to reduce the time it takes > -- > > Key: DRILL-3742 > URL: https://issues.apache.org/jira/browse/DRILL-3742 > Project: Apache Drill > Issue Type: Improvement >Reporter: Julien Le Dem > Fix For: Future > > > classpath scanning and function registry take a long time (seconds every > time). > We'd want to avoid loading the classes (use bytecode inspection instead) and > have a build time cache to avoid doing the scanning at startup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3927) Use OutOfMemoryException in more places
[ https://issues.apache.org/jira/browse/DRILL-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957937#comment-14957937 ] ASF GitHub Bot commented on DRILL-3927: --- Github user adeneche commented on the pull request: https://github.com/apache/drill/pull/199#issuecomment-148220952 +1, LGTM > Use OutOfMemoryException in more places > --- > > Key: DRILL-3927 > URL: https://issues.apache.org/jira/browse/DRILL-3927 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Chris Westin >Assignee: Chris Westin > > The new allocator uses OutOfMemoryException in better ways; some additional > exception handling sites need to catch and handle this exception in > preparation for the new allocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3936) We don't handle out of memory condition during build phase of hash join
Victoria Markman created DRILL-3936:
---
Summary: We don't handle out of memory condition during build
phase of hash join
Key: DRILL-3936
URL: https://issues.apache.org/jira/browse/DRILL-3936
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Reporter: Victoria Markman
It looks like we just fall through ( see excerpt from HashJoinBatch.java below )
{code:java}
public void executeBuildPhase() throws SchemaChangeException,
ClassTransformationException, IOException {
//Setup the underlying hash table
// skip first batch if count is zero, as it may be an empty schema batch
if (right.getRecordCount() == 0) {
for (final VectorWrapper w : right) {
w.clear();
}
rightUpstream = next(right);
}
boolean moreData = true;
while (moreData) {
switch (rightUpstream) {
case OUT_OF_MEMORY:
case NONE:
case NOT_YET:
case STOP:
moreData = false;
continue;
...
{code}
We don't handle it later either:
{code:java}
public IterOutcome innerNext() {
try {
/* If we are here for the first time, execute the build phase of the
* hash join and setup the run time generated class for the probe side
*/
if (state == BatchState.FIRST) {
// Build the hash table, using the build side record batches.
executeBuildPhase();
//IterOutcome next = next(HashJoinHelper.LEFT_INPUT,
left);
hashJoinProbe.setupHashJoinProbe(context, hyperContainer, left,
left.getRecordCount(), this, hashTable,
hjHelper, joinType);
// Update the hash table related stats for the operator
updateStats(this.hashTable);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (DRILL-3936) We don't handle out of memory condition during build phase of hash join
[
https://issues.apache.org/jira/browse/DRILL-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mehant Baid reassigned DRILL-3936:
--
Assignee: Mehant Baid
> We don't handle out of memory condition during build phase of hash join
> ---
>
> Key: DRILL-3936
> URL: https://issues.apache.org/jira/browse/DRILL-3936
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Reporter: Victoria Markman
>Assignee: Mehant Baid
>
> It looks like we just fall through ( see excerpt from HashJoinBatch.java
> below )
> {code:java}
> public void executeBuildPhase() throws SchemaChangeException,
> ClassTransformationException, IOException {
> //Setup the underlying hash table
> // skip first batch if count is zero, as it may be an empty schema batch
> if (right.getRecordCount() == 0) {
> for (final VectorWrapper w : right) {
> w.clear();
> }
> rightUpstream = next(right);
> }
> boolean moreData = true;
> while (moreData) {
> switch (rightUpstream) {
> case OUT_OF_MEMORY:
> case NONE:
> case NOT_YET:
> case STOP:
> moreData = false;
> continue;
> ...
> {code}
> We don't handle it later either:
> {code:java}
> public IterOutcome innerNext() {
> try {
> /* If we are here for the first time, execute the build phase of the
>* hash join and setup the run time generated class for the probe side
>*/
> if (state == BatchState.FIRST) {
> // Build the hash table, using the build side record batches.
> executeBuildPhase();
> //IterOutcome next = next(HashJoinHelper.LEFT_INPUT,
> left);
> hashJoinProbe.setupHashJoinProbe(context, hyperContainer, left,
> left.getRecordCount(), this, hashTable,
> hjHelper, joinType);
> // Update the hash table related stats for the operator
> updateStats(this.hashTable);
> }
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3876) flatten() should not require a subsequent project to strip columns that aren't required
[ https://issues.apache.org/jira/browse/DRILL-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957994#comment-14957994 ] ASF GitHub Bot commented on DRILL-3876: --- Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/187 > flatten() should not require a subsequent project to strip columns that > aren't required > --- > > Key: DRILL-3876 > URL: https://issues.apache.org/jira/browse/DRILL-3876 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.2.0 >Reporter: Chris Westin >Assignee: Chris Westin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3876) flatten() should not require a subsequent project to strip columns that aren't required
[ https://issues.apache.org/jira/browse/DRILL-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-3876: -- Assignee: Jason Altekruse (was: Chris Westin) > flatten() should not require a subsequent project to strip columns that > aren't required > --- > > Key: DRILL-3876 > URL: https://issues.apache.org/jira/browse/DRILL-3876 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.2.0 >Reporter: Chris Westin >Assignee: Jason Altekruse > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3876) flatten() should not require a subsequent project to strip columns that aren't required
[ https://issues.apache.org/jira/browse/DRILL-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957996#comment-14957996 ] ASF GitHub Bot commented on DRILL-3876: --- Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/187#issuecomment-148230900 I pushed the patches to master, after rebasing & run regression. I realized that the master branch is actually open while the vote for 1.2.0 is going on. > flatten() should not require a subsequent project to strip columns that > aren't required > --- > > Key: DRILL-3876 > URL: https://issues.apache.org/jira/browse/DRILL-3876 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.2.0 >Reporter: Chris Westin >Assignee: Chris Westin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-3876) flatten() should not require a subsequent project to strip columns that aren't required
[ https://issues.apache.org/jira/browse/DRILL-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni resolved DRILL-3876. --- Resolution: Fixed Fixed in commit: 6bc9111ac9ca1f87ee32a279e9c52ac06091b0a5 > flatten() should not require a subsequent project to strip columns that > aren't required > --- > > Key: DRILL-3876 > URL: https://issues.apache.org/jira/browse/DRILL-3876 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.2.0 >Reporter: Chris Westin >Assignee: Jason Altekruse > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3480) Some tpcds queries fail with with timeout errors
[ https://issues.apache.org/jira/browse/DRILL-3480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krystal updated DRILL-3480: --- Priority: Critical (was: Major) > Some tpcds queries fail with with timeout errors > > > Key: DRILL-3480 > URL: https://issues.apache.org/jira/browse/DRILL-3480 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Reporter: Krystal >Priority: Critical > Fix For: Future > > > Commit Id 9a85b2c > Some failed queries contained the following errors: > Failed while running cleanup query. Not returning connection to pool. > java.lang.InterruptedException: sleep interrupted > at java.lang.Thread.sleep(Native Method) > at > org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Channel closed /10.10.104.85:59334 <--> /10.10.104.85:31010. > Others failed with error: > org.apache.drill.common.exceptions.UserRemoteException: CONNECTION ERROR: > Exceeded timeout (4) while waiting send intermediate work fragments to > remote nodes. Sent 8 and only heard response back from 4 nodes. > [Error Id: b85205b5-3134-4f90-aca8-7d67af04f3ed] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:118) > at > org.apache.drill.exec.rpc.user.UserClient.handleReponse(UserClient.java:111) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:47) > at > org.apache.drill.exec.rpc.BasicClientWithConnection.handle(BasicClientWithConnection.java:32) > at org.apache.drill.exec.rpc.RpcBus.handle(RpcBus.java:61) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:233) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:205) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958110#comment-14958110
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42073374
--- Diff:
common/src/main/java/org/apache/drill/common/logical/LogicalPlan.java ---
@@ -94,7 +95,7 @@ public String toJsonStringSafe(DrillConfig config){
}
/** Parses a logical plan. */
- public static LogicalPlan parse(DrillConfig config, String planString) {
+ public static LogicalPlan parse(LogicalPlanPersistence config, String
planString) {
--- End diff --
wrong line?
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (DRILL-3937) We are not pruning when we have a metadata cache and auto partitioned data in some cases
Rahul Challapalli created DRILL-3937:
Summary: We are not pruning when we have a metadata cache and auto
partitioned data in some cases
Key: DRILL-3937
URL: https://issues.apache.org/jira/browse/DRILL-3937
Project: Apache Drill
Issue Type: Bug
Components: Metadata
Reporter: Rahul Challapalli
git.commit.id.abbrev=2736412
The below plan indicates that we are not pruning
{code}
explain plan for select count(*) from dfs.`/drill/comscore/orders2` where
o_clerk='Clerk#79443';
+--+--+
| text | json |
+--+--+
| 00-00Screen
00-01 Project(EXPR$0=[$0])
00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
00-03 Project($f0=[0])
00-04SelectionVectorRemover
00-05 Filter(condition=[=($0, 'Clerk#79443')])
00-06Scan(groupscan=[ParquetGroupScan
[entries=[ReadEntryWithPath
[path=maprfs:///drill/comscore/orders2/1_0_.parquet], ReadEntryWithPath
[path=maprfs:///drill/comscore/orders2/1_0_9998.parquet]],
selectionRoot=/drill/comscore/orders2, numFiles=2, usedMetadataFile=true,
columns=[`o_clerk`]]])
{code}
Error from the logs
{code}
2015-10-15 01:24:28,467 [29e0ffb4-1c91-f40a-8bf0-5e3665dcf107:foreman] WARN
o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune partition.
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to
parquet.io.api.Binary
at
org.apache.drill.exec.store.parquet.ParquetGroupScan.populatePruningVector(ParquetGroupScan.java:414)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.ParquetPartitionDescriptor.populatePartitionVectors(ParquetPartitionDescriptor.java:96)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:212)
~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
at
org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303)
[calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244)
[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
{code}
The partition column type in this case is binary which could be causing the
issue.
Partition pruning seems to be working when we have Metadata Caching + Auto
Partitioned Files with integer partition column
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3937) We are not pruning when we have a metadata cache and auto partitioned data in some cases
[
https://issues.apache.org/jira/browse/DRILL-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Challapalli updated DRILL-3937:
-
Attachment: 1_0_.parquet
1_0_9998.parquet
> We are not pruning when we have a metadata cache and auto partitioned data in
> some cases
>
>
> Key: DRILL-3937
> URL: https://issues.apache.org/jira/browse/DRILL-3937
> Project: Apache Drill
> Issue Type: Bug
> Components: Metadata
>Reporter: Rahul Challapalli
> Attachments: 1_0_9998.parquet, 1_0_.parquet
>
>
> git.commit.id.abbrev=2736412
> The below plan indicates that we are not pruning
> {code}
> explain plan for select count(*) from dfs.`/drill/comscore/orders2` where
> o_clerk='Clerk#79443';
> +--+--+
> | text | json |
> +--+--+
> | 00-00Screen
> 00-01 Project(EXPR$0=[$0])
> 00-02StreamAgg(group=[{}], EXPR$0=[COUNT()])
> 00-03 Project($f0=[0])
> 00-04SelectionVectorRemover
> 00-05 Filter(condition=[=($0, 'Clerk#79443')])
> 00-06Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=maprfs:///drill/comscore/orders2/1_0_.parquet], ReadEntryWithPath
> [path=maprfs:///drill/comscore/orders2/1_0_9998.parquet]],
> selectionRoot=/drill/comscore/orders2, numFiles=2, usedMetadataFile=true,
> columns=[`o_clerk`]]])
> {code}
> Error from the logs
> {code}
> 2015-10-15 01:24:28,467 [29e0ffb4-1c91-f40a-8bf0-5e3665dcf107:foreman] WARN
> o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune
> partition.
> java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to
> parquet.io.api.Binary
> at
> org.apache.drill.exec.store.parquet.ParquetGroupScan.populatePruningVector(ParquetGroupScan.java:414)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.ParquetPartitionDescriptor.populatePartitionVectors(ParquetPartitionDescriptor.java:96)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.PruneScanRule.doOnMatch(PruneScanRule.java:212)
> ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.logical.partition.ParquetPruneScanRule$2.onMatch(ParquetPruneScanRule.java:87)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
> [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
> [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
> [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:303)
> [calcite-core-1.4.0-drill-r6.jar:1.4.0-drill-r6]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:545)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:213)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:248)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:905)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:244)
> [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> {code}
> The partition column type in this case is binary which could be causing the
> issue.
> Partition pruning seems to be working when we have Metadata Caching + Auto
> Partitioned Files with integer partition column
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2808) Substr function throws weird message when wrong arguments are passed in
[
https://issues.apache.org/jira/browse/DRILL-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958270#comment-14958270
]
ASF GitHub Bot commented on DRILL-2808:
---
GitHub user hsuanyi opened a pull request:
https://github.com/apache/drill/pull/204
DRILL-2808: In constant folding, when the the data type of the return…
… value is TimeStamp, generate TimeStamp Literal
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hsuanyi/incubator-drill DRILL-2808
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/204.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #204
commit a1844e132435238e995c7d1b8ef8a4641c415a29
Author: Hsuan-Yi Chu
Date: 2015-10-15T03:55:34Z
DRILL-2808: In constant folding, when the the data type of the return value
is TimeStamp, generate TimeStamp Literal
> Substr function throws weird message when wrong arguments are passed in
> ---
>
> Key: DRILL-2808
> URL: https://issues.apache.org/jira/browse/DRILL-2808
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
>Affects Versions: 0.9.0
>Reporter: Victoria Markman
> Fix For: Future
>
>
> #Wed Apr 15 17:37:25 EDT 2015
> git.commit.id.abbrev=cb47df0
> {code}
> 0: jdbc:drill:schema=dfs> select substr(0, 1, 'abc') from sys.options limit 1;
> Query failed: SYSTEM ERROR: Unexpected exception during fragment
> initialization: Internal error: Error while applying rule
> ReduceExpressionsRule[Project], args
> [rel#53990:ProjectRel.NONE.ANY([]).[](child=rel#53989:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> [1f7e0c35-fc65-4a2f-b668-12e9baf5c9b8 on atsqa4-133.qa.lab:31010]
> Error: exception while executing query: Failure while executing query.
> (state=,code=0)
> {code}
> drillbit.log
> {code}
> 2015-04-16 22:04:32,029 [2acfce0f-8af4-9786-4799-9180b9bac219:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - State change requested. PENDING -->
> FAILED
> org.apache.drill.exec.work.foreman.ForemanException: Unexpected exception
> during fragment initialization: Internal error: Error while applying rule
> ReduceExpressionsRule[Project], args
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:211)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.lang.AssertionError: Internal error: Error while applying
> rule ReduceExpressionsRule[Project], args
> [rel#54047:ProjectRel.NONE.ANY([]).[](child=rel#54046:Subset#0.ENUMERABLE.ANY([]).[],EXPR$0=SUBSTR(0,
> 1, 'abc'))]
> at org.eigenbase.util.Util.newInternal(Util.java:750)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:246)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.eigenbase.relopt.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:661)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> net.hydromatic.optiq.tools.Programs$RuleSetProgram.run(Programs.java:165)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> net.hydromatic.optiq.prepare.PlannerImpl.transform(PlannerImpl.java:278)
> ~[optiq-core-0.9-drill-r21.jar:na]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:196)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:137)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:155)
> ~[drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:770)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:202)
> [drill-java-exec-0.9.0-SNAPSHOT-rebuffed.jar:0.9.0-SNAPSHOT]
> ... 3 common frames omitted
> Caused by: java.lang.RuntimeException: Error in evaluating function of
> castBIGINT
> at
> org.apache.drill.exec.expr.fn.inte
[jira] [Commented] (DRILL-3769) to_date function with one argument returns wrong data type
[
https://issues.apache.org/jira/browse/DRILL-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958284#comment-14958284
]
ASF GitHub Bot commented on DRILL-3769:
---
GitHub user hsuanyi opened a pull request:
https://github.com/apache/drill/pull/205
DRILL-3769: Allow to_date() to use castDate()'s implementations
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hsuanyi/incubator-drill DRILL-3769
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/205.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #205
commit 639dcd7bb56483bb2e6ac0e1adbd918d7dd850e5
Author: Hsuan-Yi Chu
Date: 2015-10-13T05:52:05Z
DRILL-3769: Allow to_date() to use castDate()'s implementations
> to_date function with one argument returns wrong data type
> --
>
> Key: DRILL-3769
> URL: https://issues.apache.org/jira/browse/DRILL-3769
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.1.0, 1.2.0
>Reporter: Victoria Markman
> Fix For: Future
>
>
> 1. to_date function is not part of SQL standard according to my research
> (checked ISO/IEC9075-2), so implementations of it may vary from database to
> database (our implementation of to_date is different from Postgres, for
> example)
> 2. Our documentation only talks about to_date with 2 parameters: format and
> actual string to be converted to date type
> 3. Calcite does not seem to have to_date, which makes me think that this is
> Drill UDF
> 4. Apparently, if you invoke to_date() with one argument in Drill: it runs.
>
> So there are two possibilities: we implemented to_date with one argument to
> be compatible with some other SQL engine, Hive ?
>or
>it is a bug and we should throw an error.
> You can use to_date with one argument in a simple query:
> {code}
> 0: jdbc:drill:schema=dfs> select to_date(c1 + interval '1' day) from t1 limit
> 1;
> +-+
> | EXPR$0|
> +-+
> | 2015-01-02 |
> +-+
> 1 row selected (0.242 seconds)
> {code}
> However, since return type is varbinary, joins, aggregations and CTAS are
> going to be problematic.
> Here is to_date use in join to illustrate this (c1 is a date column):
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1, t2 where to_date(t1.c1) = t2.c2;
> Error: SYSTEM ERROR: DrillRuntimeException: Join only supports implicit casts
> between 1. Numeric data
> 2. Varchar, Varbinary data 3. Date, Timestamp data Left type: DATE, Right
> type: VAR16CHAR. Add explicit casts to avoid this error
> Fragment 0:0
> [Error Id: 66ac8248-56c5-401a-aa53-de90cb828bc4 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Since we don't support cast between varbinary and date, attempt to cast it
> results in:
> {code}
> 0: jdbc:drill:schema=dfs> select * from t1, t2 where cast(to_date(t1.c1) as
> date) = t2.c2;
> Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to
> materialize incoming schema. Errors:
> Error in expression at index -1. Error: Missing function implementation:
> [castBIGINT(VAR16CHAR-OPTIONAL)]. Full expression: --UNKNOWN EXPRESSION--..
> Fragment 0:0
> [Error Id: deeb040a-f1d3-4ea0-8849-7ced29508576 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Same with CTAS:
> {code}
> 0: jdbc:drill:schema=dfs> create table x(a1) as select to_date(c1) from t1;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 10 |
> +---++
> 1 row selected (0.4 seconds)
> 0: jdbc:drill:schema=dfs> select * from x;
> +--+
> | a1 |
> +--+
> | [B@28b5395d |
> | [B@11c91d8c |
> | [B@2ab2db73 |
> | [B@446570eb |
> | [B@5fd87761 |
> | [B@7c85b26f |
> | [B@2d85d547 |
> | [B@2d753faa |
> | null |
> | [B@6ca6c936 |
> +--+
> 10 rows selected (0.183 seconds)
> 0: jdbc:drill:schema=dfs> select cast(a1 as date) from x;
> Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must
> be in the range [1,12]
> Fragment 0:0
> [Error Id: 71d8cd8f-6c88-4a13-9d24-b06ef52f6572 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
[
https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958413#comment-14958413
]
ASF GitHub Bot commented on DRILL-3742:
---
Github user julienledem commented on a diff in the pull request:
https://github.com/apache/drill/pull/148#discussion_r42087783
--- Diff: contrib/storage-hive/core/pom.xml ---
@@ -114,36 +114,17 @@
2.3.19
-
- src/main/codegen/config.fmpp
- target/generated-sources
- src/main/codegen/templates
-
generate-fmpp-sources
-initialize
+generate-sources
generate
-
-
-
-
-org.codehaus.mojo
-build-helper-maven-plugin
-1.5
-
-
-add-fmpp-sources
-validate
-
- add-source
-
-
-
${project.build.directory}/generated-sources
-
+ src/main/codegen/config.fmpp
+ target/generated-sources
--- End diff --
s/target/${build root}
> Improve classpath scanning to reduce the time it takes
> --
>
> Key: DRILL-3742
> URL: https://issues.apache.org/jira/browse/DRILL-3742
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Julien Le Dem
> Fix For: Future
>
>
> classpath scanning and function registry take a long time (seconds every
> time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and
> have a build time cache to avoid doing the scanning at startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
