[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459790#comment-15459790
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77421538
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
NESTED_LOOP_JOIN = 35;
AVRO_SUB_SCAN = 36;
}
+
+message Func {
+ optional string name = 1;
+ repeated common.MajorType major_type = 2;
--- End diff --
What does major type mean for a function? Is this the parameter type? The
return type? Something else? Perhaps choose a clearer name or include a comment
to explain the meaning.
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459787#comment-15459787
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77421468
--- Diff: protocol/src/main/protobuf/UserBitShared.proto ---
@@ -298,3 +298,17 @@ enum CoreOperatorType {
NESTED_LOOP_JOIN = 35;
AVRO_SUB_SCAN = 36;
}
+
+message Func {
--- End diff --
Protobufs are hard to guess at without documentation; there is no code to
provide a hint about the meaning of fields. Can you provide a description of
how these are used?
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459778#comment-15459778
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77421120
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
debug: {
return_error_for_failure_in_cancelled_fragments: false
}
+ udf: {
+retry.times: 5,
+directory: {
+ base: "/tmp",
--- End diff --
Is there a /tmp in DFS? If so, this would be a shared resoure. Should the
base be /tmp/drill to keep our temp files distinct from those of other apps?
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459774#comment-15459774
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77421023
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -189,6 +189,15 @@ drill.exec: {
debug: {
return_error_for_failure_in_cancelled_fragments: false
}
+ udf: {
+retry.times: 5,
+directory: {
+ base: "/tmp",
+ staging: ${drill.exec.zk.root}"/udf/staging",
--- End diff --
zk.root is the root zNode in ZK; it is not a file system node in DFS.
See above comment about defining a new drill.dfs.root property.
I like how you've used the ${} substitution here.
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459452#comment-15459452
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77403990
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java
---
@@ -141,25 +141,41 @@ public void validate(final OptionValue v, final
OptionManager manager) {
public static class BooleanValidator extends TypeValidator {
public BooleanValidator(String name, boolean def) {
- super(name, Kind.BOOLEAN,
OptionValue.createBoolean(OptionType.SYSTEM, name, def));
+ this(name, def, false);
+}
+
+public BooleanValidator(String name, boolean def, boolean
isAdminOption) {
+ super(name, Kind.BOOLEAN,
OptionValue.createBoolean(OptionType.SYSTEM, name, def), isAdminOption);
}
}
public static class StringValidator extends TypeValidator {
public StringValidator(String name, String def) {
- super(name, Kind.STRING, OptionValue.createString(OptionType.SYSTEM,
name, def));
+ this(name, def, false);
+}
+
+public StringValidator(String name, String def, boolean isAdminOption)
{
--- End diff --
See above comment.
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459450#comment-15459450
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77403860
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/TypeValidators.java
---
@@ -141,25 +141,41 @@ public void validate(final OptionValue v, final
OptionManager manager) {
public static class BooleanValidator extends TypeValidator {
public BooleanValidator(String name, boolean def) {
- super(name, Kind.BOOLEAN,
OptionValue.createBoolean(OptionType.SYSTEM, name, def));
+ this(name, def, false);
+}
+
+public BooleanValidator(String name, boolean def, boolean
isAdminOption) {
--- End diff --
Must be auto-generated code... Perhaps change "isAdminOption" to just
"option" as a Boolean validator would seem to want to work with any boolean
option...
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459444#comment-15459444
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77403503
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/OptionValidator.java
---
@@ -69,6 +75,13 @@ public int getTtl() {
}
/**
+ * @return true is option can be set on system level only
--- End diff --
Explain a bit better. What is meant by "on system level only"? That this
can be set only for the whole Drill system? That this is a system-level
property that only the admin (not user) can set?
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459436#comment-15459436
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77403174
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDropFunction.java
---
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DropFunctionHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.List;
+
+public class SqlDropFunction extends DrillSqlCall {
+
+ private final SqlNode jar;
--- End diff --
The "boilerplate" is the same between this class and SqlCreateFunction. Is
it worth factoring that stuff out into a common base class?
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459434#comment-15459434
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77403003
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlDropFunction.java
---
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.DropFunctionHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.List;
+
+public class SqlDropFunction extends DrillSqlCall {
+
+ private final SqlNode jar;
+
+ public static final SqlSpecialOperator OPERATOR = new
SqlSpecialOperator("DROP_FUNCTION", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, SqlParserPos
pos, SqlNode... operands) {
+ return new SqlDropFunction(pos, operands[0]);
+}
+ };
+
+ public SqlDropFunction(SqlParserPos pos, SqlNode jar) {
+super(pos);
+this.jar = jar;
+ }
+
+ @Override
+ public SqlOperator getOperator() {
+return OPERATOR;
+ }
+
+ @Override
+ public List getOperandList() {
+List opList = Lists.newArrayList();
--- End diff --
See comment above.
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459430#comment-15459430
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77402886
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/SqlCreateFunction.java
---
@@ -0,0 +1,79 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.parser;
+
+import com.google.common.collect.Lists;
+import org.apache.calcite.sql.SqlCall;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.SqlLiteral;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.SqlOperator;
+import org.apache.calcite.sql.SqlSpecialOperator;
+import org.apache.calcite.sql.SqlWriter;
+import org.apache.calcite.sql.parser.SqlParserPos;
+import org.apache.drill.exec.planner.sql.handlers.AbstractSqlHandler;
+import org.apache.drill.exec.planner.sql.handlers.CreateFunctionHandler;
+import org.apache.drill.exec.planner.sql.handlers.SqlHandlerConfig;
+
+import java.util.List;
+
+public class SqlCreateFunction extends DrillSqlCall {
+
+ private final SqlNode jar;
+
+ public static final SqlSpecialOperator OPERATOR = new
SqlSpecialOperator("CREATE_FUNCTION", SqlKind.OTHER) {
+@Override
+public SqlCall createCall(SqlLiteral functionQualifier, SqlParserPos
pos, SqlNode... operands) {
+ return new SqlCreateFunction(pos, operands[0]);
+}
+ };
+
+ public SqlCreateFunction(SqlParserPos pos, SqlNode jar) {
+super(pos);
+this.jar = jar;
+ }
+
+ @Override
+ public SqlOperator getOperator() {
+return OPERATOR;
+ }
+
+ @Override
+ public List getOperandList() {
+List opList = Lists.newArrayList();
--- End diff --
Java code says that the newArrayList( ) technique is obsolete. With
parameterized types, the preferred form is now:
List opList = new ArrayList<>( );
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459419#comment-15459419 ] ASF GitHub Bot commented on DRILL-4726: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/574#discussion_r77402584 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java --- @@ -106,11 +105,19 @@ String RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS = "drill.exec.debug.return_error_for_failure_in_cancelled_fragments"; + String CLIENT_SUPPORT_COMPLEX_TYPES = "drill.client.supports-complex-types"; + /** + * Configuration properties connected with dynamic UDFs support + */ --- End diff -- Are these only for dynamic UDFs? Do we have settings for (non-dynamic) UDFs? If so, should we have drill.exec.udf prop1 dynamic prop1 prop2 > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459413#comment-15459413 ] ASF GitHub Bot commented on DRILL-4726: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/574#discussion_r77402402 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java --- @@ -106,11 +105,19 @@ String RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS = "drill.exec.debug.return_error_for_failure_in_cancelled_fragments"; + String CLIENT_SUPPORT_COMPLEX_TYPES = "drill.client.supports-complex-types"; + /** + * Configuration properties connected with dynamic UDFs support + */ + String UDF_RETRY_TIMES = "drill.exec.udf.retry.times"; --- End diff -- Since retry is not a group, just use retry-times (or retry_times). > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[
https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459311#comment-15459311
]
ASF GitHub Bot commented on DRILL-4726:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/574#discussion_r77396445
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java ---
@@ -106,11 +105,19 @@
String RETURN_ERROR_FOR_FAILURE_IN_CANCELLED_FRAGMENTS =
"drill.exec.debug.return_error_for_failure_in_cancelled_fragments";
+ String CLIENT_SUPPORT_COMPLEX_TYPES =
"drill.client.supports-complex-types";
+ /**
+ * Configuration properties connected with dynamic UDFs support
+ */
+ String UDF_RETRY_TIMES = "drill.exec.udf.retry.times";
+ String UDF_DIRECTORY_STAGING = "drill.exec.udf.directory.staging";
+ String UDF_DIRECTORY_REGISTRY = "drill.exec.udf.directory.registry";
+ String UDF_DIRECTORY_TMP = "drill.exec.udf.directory.tmp";
+ String UDF_DIRECTORY_BASE = "drill.exec.udf.directory.base";
+ String UDF_DIRECTORY_FS = "drill.exec.udf.directory.fs";
--- End diff --
Let's discuss. DoY uses DFS to store files. Let's define a general property
drill.dfs.home that points to the root DFS home folder. This can default to
"/user/drill" for HDFS. ("/user" is an HDFS standard.) Users can change the
root in a single place. Then, when merging DoY, we'll use the new
drill.dfs.home property in place of the one that DoY currently uses.
Note also that Typesafe allows property values of the form:
drill.exec.udf.directory.fs: "$drill.dfs.home/udf" to make it easy to
compute derived paths.
> Dynamic UDFs support
>
>
> Key: DRILL-4726
> URL: https://issues.apache.org/jira/browse/DRILL-4726
> Project: Apache Drill
> Issue Type: New Feature
>Affects Versions: 1.6.0
>Reporter: Arina Ielchiieva
>Assignee: Arina Ielchiieva
> Fix For: Future
>
>
> Allow register UDFs without restart of Drillbits.
> Design is described in document below:
> https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459293#comment-15459293 ] ASF GitHub Bot commented on DRILL-4726: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/574#discussion_r77395696 --- Diff: exec/java-exec/src/main/codegen/data/Parser.tdd --- @@ -38,7 +38,8 @@ "REFRESH", "METADATA", "DATABASE", -"IF" +"IF", +"JAR" --- End diff -- Should these be alphabetical? > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459292#comment-15459292 ] ASF GitHub Bot commented on DRILL-4726: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/574#discussion_r77395651 --- Diff: distribution/src/resources/sqlline.bat --- @@ -114,6 +114,10 @@ if "test%DRILL_LOG_DIR%" == "test" ( set DRILL_LOG_DIR=%DRILL_HOME%\log ) +if "test%DRILL_UDF_DIR%" == "test" ( + set DRILL_UDF_DIR=%DRILL_HOME%\udf +) + --- End diff -- %DRILL_CONF_DIR%\udf See comment above. > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459287#comment-15459287 ] ASF GitHub Bot commented on DRILL-4726: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/574#discussion_r77395469 --- Diff: distribution/src/resources/drill-config.sh --- @@ -366,11 +366,18 @@ if [ $? -eq 0 ]; then fatal_error "Java 1.7 or later is required to run Apache Drill." fi +# Create Drill local udf area, if it does not exist +DRILL_UDF_DIR="$DRILL_HOME/udf" +if [[ ! -d "$DRILL_UDF_DIR" ]]; then + mkdir -p "$DRILL_UDF_DIR" +fi + # Adjust paths for CYGWIN if $is_cygwin; then DRILL_HOME=`cygpath -w "$DRILL_HOME"` DRILL_CONF_DIR=`cygpath -w "$DRILL_CONF_DIR"` DRILL_LOG_DIR=`cygpath -w "$DRILL_LOG_DIR"` + DRILL_UDF_DIR=`cygpath -w "$DRILL_UDF_DIR"` --- End diff -- Do this only if a DRILL_UDF_DIR exists. Actually, not sure if this is even needed if we compute $DRILL_UDF_DIR as $DRILL_CONF_DIR/udf. > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459284#comment-15459284 ] ASF GitHub Bot commented on DRILL-4726: --- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/574#discussion_r77395331 --- Diff: distribution/src/resources/drill-config.sh --- @@ -366,11 +366,18 @@ if [ $? -eq 0 ]; then fatal_error "Java 1.7 or later is required to run Apache Drill." fi +# Create Drill local udf area, if it does not exist +DRILL_UDF_DIR="$DRILL_HOME/udf" +if [[ ! -d "$DRILL_UDF_DIR" ]]; then + mkdir -p "$DRILL_UDF_DIR" +fi + --- End diff -- This should be $DRILL_CONF_DIR/udf, not $DRILL_HOME. $DRILL_CONF_DIR points to the user's "site" directory, which will be outside of $DRILL_HOME for Drill-on-YARN. Not sure we need to create this directory. The pattern followed elsewhere is to include the directory on the class path if it exists. In Drill-on-YARN, we CAN'T create the directory; the "localized" Drill and site directories are read-only. > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432485#comment-15432485 ] Arina Ielchiieva commented on DRILL-4726: - Link to PR - https://github.com/apache/drill/pull/574 > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1FfyJtWae5TLuyheHCfldYUpCdeIezR2RlNsrOTYyAB4/edit?usp=sharing > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4726) Dynamic UDFs support
[ https://issues.apache.org/jira/browse/DRILL-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338215#comment-15338215 ] Paul Rogers commented on DRILL-4726: Added many comments in the doc and on the user mailing list of the form "have we considered problem X?" Here are a few references for possible solutions. Hive is one good model. Hive adds code using the Hive CLI (equivelent to Drill's sqlline client). It seems that code can be added globally, or per session. (Per-session would be handy in Drill for data exporation that needs a one-off function without the need to install the code for all users.) ADD JAR your.jar; Adds a jar to the class path. Something similar in Drill would allow adding any number of jar files, including dependencies. Then: CREATE TEMPORARY FUNCTION myFunc AS 'com.yoyodyne.MyFunc'; This does three things. First, it maps a SQL name to the function. Second, it allows any number of functions in the same jar. And third, it establishes a life cycle for the function (used only for this one section in the above example.) Hive also provides; LIST JARS; To identify what is on the class path. This command implies that Hive has a separate class path (class loader) for user jars, since the command does not list Hive's own jars and dependencies. For Drill, we'd want to list jars added by the user, not the many, many jars that ship with Drill. Other details: DELETE TEMPORARY FUNCTION myFunc; DELETE JAR your.jar; To remove a function (1) or jar. Interestingly, the jar commands are implemented by the CLI tool, the function commands by the Hive engine. Hive uses the idea of Resources (2) to distribute the code for each job. Hive supports other resources such as files and archives. (3) Hive executes a query using MapReduce, which makes it easier to distribute code for each job. With Drill we'd need to invent a different mechanism. Of course, all of this may be well beyond the scope of this JIRA. But, it does provide a perspective of what other products do. (1) https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/Drop/ReloadFunction (2) https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-HiveResources for information on Hive resources (3) https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Commands > Dynamic UDFs support > > > Key: DRILL-4726 > URL: https://issues.apache.org/jira/browse/DRILL-4726 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.6.0 >Reporter: Arina Ielchiieva >Assignee: Arina Ielchiieva > Fix For: Future > > > Allow register UDFs without restart of Drillbits. > Design is described in document below: > https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
