[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715753#comment-16715753 ] Bridget Bevens commented on DRILL-3988: --- Hi [~kkhatua] and [~arina], I've updated the following doc to include changes listed in this JIRA. Please have a look and let me know if I need to change anything. [https://drill.apache.org/docs/querying-system-tables/] [https://drill.apache.org/docs/querying-system-tables/#querying-the-functions-table] Thanks, Bridget > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-complete > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646797#comment-16646797
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua closed pull request #1483: DRILL-3988: Expose Drill built-in functions
& UDFs in a system table
URL: https://github.com/apache/drill/pull/1483
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
index f4b83736739..a6f9a7f7ca7 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
@@ -28,6 +28,7 @@
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
@@ -54,6 +55,7 @@
import org.apache.drill.exec.exception.FunctionValidationException;
import org.apache.drill.exec.exception.JarValidationException;
import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
import org.apache.drill.exec.expr.fn.registry.JarScan;
import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
import org.apache.drill.exec.planner.sql.DrillOperatorTable;
@@ -483,6 +485,17 @@ private ScanResult scan(ClassLoader classLoader, Path
path, URL[] urls) throws I
return missingJars;
}
+ /**
+ * Retrieve all functions, mapped by source jars (after syncing)
+ * @return Map of source jars and their functionHolders
+ */
+ public Map> getAllJarsWithFunctionsHolders() {
+if (useDynamicUdfs) {
+ syncWithRemoteRegistry(localFunctionRegistry.getVersion());
+}
+return localFunctionRegistry.getAllJarsWithFunctionsHolders();
+ }
+
/**
* Creates local udf directory, if it doesn't exist.
* Checks if local udf directory is a directory and if current application
has write rights on it.
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
index d1d4fc94dfd..d0383e91fc8 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
@@ -25,6 +25,8 @@
import org.apache.drill.exec.expr.fn.DrillFuncHolder;
import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Queue;
@@ -161,6 +163,38 @@ public void removeJar(String jarName) {
}
}
+ /**
+ * Retrieves all functions (holders) associated with all the jars
+ * This is read operation, so several users can perform this operation at
the same time.
+ * @return list of all functions, mapped by their sources
+ */
+ public Map> getAllJarsWithFunctionHolders() {
+Map> allFunctionHoldersByJar = new
HashMap<>();
+
+try (@SuppressWarnings("unused") Closeable lock = readLock.open()) {
+ for (String jarName : jars.keySet()) {
+//Capture functionHolders here
+List drillFuncHolderList = new LinkedList<>();
+
+Map> functionsInJar = jars.get(jarName);
+for (Map.Entry> functionEntry :
functionsInJar.entrySet()) {
+ String fnName = functionEntry.getKey();
+ Queue fnSignatureList = functionEntry.getValue();
+ //Get all FunctionHolders (irrespective of source)
+ Map functionHolders = functions.get(fnName);
+ //Iterate for matching entries and populate new Map
+ for (Map.Entry entry :
functionHolders.entrySet()) {
+if (fnSignatureList.contains(entry.getKey())) {
+ drillFuncHolderList.add(new FunctionHolder(fnName,
entry.getKey(), entry.getValue()));
+}
+ }
+}
+allFunctionHoldersByJar.put(jarName, drillFuncHolderList);
+ }
+}
+return allFunctionHoldersByJar;
+ }
+
/**
* Retrieves all function names associated with the jar from {@link #jars}.
* Returns empty list if jar is not registered.
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/LocalFunctionRegistry.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/LocalFunctionRegistry.java
index
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646793#comment-16646793
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224535223
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolderTest.java
##
@@ -155,6 +162,47 @@ public void testGetAllJarNames() {
compareTwoLists(expectedResult, registryHolder.getAllJarNames());
}
+ @Test
+ public void testGetAllJarsWithFunctionHolders() {
+Map> fnHoldersInRegistry =
registryHolder.getAllJarsWithFunctionHolders();
+//Iterate and confirm lists are same
+for (String jarName : newJars.keySet()) {
+ List expectedHolderList = newJars.get(jarName).stream()
+ .map(FunctionHolder::getHolder) //Extract DrillFuncHolder
+ .collect(Collectors.toList());
+ List testHolderList =
fnHoldersInRegistry.get(jarName).stream()
+ .map(FunctionHolder::getHolder) //Extract DrillFuncHolder
+ .collect(Collectors.toList());
+
+ compareTwoLists(expectedHolderList, testHolderList);
+}
+
+Map shuffleFunctionMap = new HashMap<>();
+// Confirm that same function spans multiple jars with different signatures
+//Init: Expected Map of items
+for (String jarName : newJars.keySet()) {
+ for (FunctionHolder funcHolder : newJars.get(jarName)) {
+if (funcHolder.getName().equals(SHUFFLE_FUNC_NAME)) {
Review comment:
👍
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646220#comment-16646220
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224387658
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolderTest.java
##
@@ -155,6 +162,47 @@ public void testGetAllJarNames() {
compareTwoLists(expectedResult, registryHolder.getAllJarNames());
}
+ @Test
+ public void testGetAllJarsWithFunctionHolders() {
+Map> fnHoldersInRegistry =
registryHolder.getAllJarsWithFunctionHolders();
+//Iterate and confirm lists are same
+for (String jarName : newJars.keySet()) {
+ List expectedHolderList = newJars.get(jarName).stream()
+ .map(FunctionHolder::getHolder) //Extract DrillFuncHolder
+ .collect(Collectors.toList());
+ List testHolderList =
fnHoldersInRegistry.get(jarName).stream()
+ .map(FunctionHolder::getHolder) //Extract DrillFuncHolder
+ .collect(Collectors.toList());
+
+ compareTwoLists(expectedHolderList, testHolderList);
+}
+
+Map shuffleFunctionMap = new HashMap<>();
+// Confirm that same function spans multiple jars with different signatures
+//Init: Expected Map of items
+for (String jarName : newJars.keySet()) {
+ for (FunctionHolder funcHolder : newJars.get(jarName)) {
+if (funcHolder.getName().equals(SHUFFLE_FUNC_NAME)) {
Review comment:
General recommendation is not use equals on constant to avoid NPE:
`SHUFFLE_FUNC_NAME.equals(funcHolder.getName())`. Please change here and in the
code below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645640#comment-16645640
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224263847
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolderTest.java
##
@@ -202,26 +250,38 @@ public void testGetAllFunctionsWithSignatures() {
public void testGetHoldersByFunctionNameWithVersion() {
List expectedResult = newJars.values().stream()
.flatMap(Collection::stream)
-.filter(f -> "lower".equals(f.getName()))
+.filter(f -> LOWER_FUNC_NAME.equals(f.getName()))
.map(FunctionHolder::getHolder)
.collect(Collectors.toList());
assertFalse(expectedResult.isEmpty());
AtomicInteger version = new AtomicInteger();
-compareTwoLists(expectedResult,
registryHolder.getHoldersByFunctionName("lower", version));
+compareTwoLists(expectedResult,
registryHolder.getHoldersByFunctionName(LOWER_FUNC_NAME, version));
assertEquals("Version number should match", version.get(),
registryHolder.getVersion());
}
@Test
public void testGetHoldersByFunctionName() {
-List expectedResult = newJars.values().stream()
-.flatMap(Collection::stream)
-.filter(f -> "lower".equals(f.getName()))
-.map(FunctionHolder::getHolder)
-.collect(Collectors.toList());
+List expectedUniqueResult = Lists.newArrayList();
Review comment:
Done. removed all references to Lists and removed the Guava import as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645638#comment-16645638
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224263558
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolderTest.java
##
@@ -155,6 +162,47 @@ public void testGetAllJarNames() {
compareTwoLists(expectedResult, registryHolder.getAllJarNames());
}
+ @Test
+ public void testGetAllJarsWithFunctionHolders() {
+Map> fnHoldersInRegistry =
registryHolder.getAllJarsWithFunctionHolders();
+//Iterate and confirm lists are same
+for (String jarName : Lists.newArrayList(newJars.keySet())) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645623#comment-16645623
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224259705
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
##
@@ -161,6 +163,38 @@ public void removeJar(String jarName) {
}
}
+ /**
+ * Retrieves all functions (holders) associated with all the jars
+ * This is read operation, so several users can perform this operation at
the same time.
+ * @return list of all functions, mapped by their sources
+ */
+ public Map> getAllJarsWithFunctionHolders() {
+Map> allFunctionHoldersByJar = new
HashMap<>();
+
+try (@SuppressWarnings("unused") Closeable lock = readLock.open()) {
+ for (String jarName : jars.keySet()) {
+//Capture functionHolders here
+List drillFuncHolderList = new LinkedList<>();
Review comment:
Since the variable is only used to add and remove the function holders, this
seemed most efficient for memory and speed. Some sources (like `built-in`) will
have 2K+ functions, while UDFs will have as little as 1 function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645615#comment-16645615
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224258892
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
Review comment:
Ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645614#comment-16645614
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224258810
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645393#comment-16645393
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224193497
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
Review comment:
My bad. I was switching between this (list) and a map when implementing, so
I missed removing this during refactoring. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645385#comment-16645385
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224191496
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ for (FunctionHolder dfhEntry : jarFunctionListMap.get(jarName)) {
+populateFunctionMap(functionMap, jarName, dfhEntry.getHolder());
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort((FunctionInfo o1, FunctionInfo o2) -> {
+ int result = o1.name.compareTo(o2.name);
+ if (result == 0) {
+result = o1.signature.compareTo(o2.signature);
+ }
+ if (result == 0) {
+return o1.returnType.compareTo(o2.returnType);
+ }
+ return result;
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ /**
+ * Populate the map of functionInfo based on the functionSignatureKey for a
given Jar-DrillFunctionHolder
+ * @param functionMap map to populate
+ * @param jarName name of the source jar
+ * @param dfh functionHolder that carries all the registered names and the
signature
+ */
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a unique key for a function holder as
'functionName#functionSignature'
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
+ functionMap.put(funcSignatureKey, new FunctionInfo(name, signature,
returnType, jarName));
+}
+ }
+
+ @Override
+ public boolean hasNext() {
+return sortedIterator.hasNext();
+ }
+
+ @Override
+ public FunctionInfo next() {
+return sortedIterator.next();
+ }
+
+ /**
+ * Representation of an entry in the System table - Functions
+ */
+ public static class FunctionInfo {
+@NonNullable
+public final String name;
+@NonNullable
+public final String signature;
+public final String returnType;
Review comment:
StringBuilder capacity was made 64 because the default capacity is 16, which
would almost always be insufficient, forcing a resize. 64 seemed like a
reasonable value. I'll add a comment to explain.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries a
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645381#comment-16645381
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224190591
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ for (FunctionHolder dfhEntry : jarFunctionListMap.get(jarName)) {
+populateFunctionMap(functionMap, jarName, dfhEntry.getHolder());
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort((FunctionInfo o1, FunctionInfo o2) -> {
+ int result = o1.name.compareTo(o2.name);
+ if (result == 0) {
+result = o1.signature.compareTo(o2.signature);
+ }
+ if (result == 0) {
+return o1.returnType.compareTo(o2.returnType);
+ }
+ return result;
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ /**
+ * Populate the map of functionInfo based on the functionSignatureKey for a
given Jar-DrillFunctionHolder
+ * @param functionMap map to populate
+ * @param jarName name of the source jar
+ * @param dfh functionHolder that carries all the registered names and the
signature
+ */
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a unique key for a function holder as
'functionName#functionSignature'
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
+ functionMap.put(funcSignatureKey, new FunctionInfo(name, signature,
returnType, jarName));
+}
+ }
+
+ @Override
+ public boolean hasNext() {
+return sortedIterator.hasNext();
+ }
+
+ @Override
+ public FunctionInfo next() {
+return sortedIterator.next();
+ }
+
+ /**
+ * Representation of an entry in the System table - Functions
+ */
+ public static class FunctionInfo {
+@NonNullable
+public final String name;
+@NonNullable
+public final String signature;
+public final String returnType;
Review comment:
For source, yes. But I was not sure if all functions must return a value. I
guess, as a SQL function, the function is expected to return something. Will
apply the tag
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contac
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644745#comment-16644745
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224006820
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolderTest.java
##
@@ -155,6 +162,47 @@ public void testGetAllJarNames() {
compareTwoLists(expectedResult, registryHolder.getAllJarNames());
}
+ @Test
+ public void testGetAllJarsWithFunctionHolders() {
+Map> fnHoldersInRegistry =
registryHolder.getAllJarsWithFunctionHolders();
+//Iterate and confirm lists are same
+for (String jarName : Lists.newArrayList(newJars.keySet())) {
Review comment:
No need to wrap into ArrayList here and a couple more times below in the
code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644739#comment-16644739
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224004538
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
Review comment:
Why `StringBuilder` capacity is 64?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644741#comment-16644741
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224004849
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ for (FunctionHolder dfhEntry : jarFunctionListMap.get(jarName)) {
+populateFunctionMap(functionMap, jarName, dfhEntry.getHolder());
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort((FunctionInfo o1, FunctionInfo o2) -> {
+ int result = o1.name.compareTo(o2.name);
+ if (result == 0) {
+result = o1.signature.compareTo(o2.signature);
+ }
+ if (result == 0) {
+return o1.returnType.compareTo(o2.returnType);
+ }
+ return result;
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ /**
+ * Populate the map of functionInfo based on the functionSignatureKey for a
given Jar-DrillFunctionHolder
+ * @param functionMap map to populate
+ * @param jarName name of the source jar
+ * @param dfh functionHolder that carries all the registered names and the
signature
+ */
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a unique key for a function holder as
'functionName#functionSignature'
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
+ functionMap.put(funcSignatureKey, new FunctionInfo(name, signature,
returnType, jarName));
+}
+ }
+
+ @Override
+ public boolean hasNext() {
+return sortedIterator.hasNext();
+ }
+
+ @Override
+ public FunctionInfo next() {
+return sortedIterator.next();
+ }
+
+ /**
+ * Representation of an entry in the System table - Functions
+ */
+ public static class FunctionInfo {
+@NonNullable
+public final String name;
+@NonNullable
+public final String signature;
+public final String returnType;
Review comment:
Ypu have mentioned that issue with return type if fixed, should it be
`@NonNullable`? the same for source.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Crea
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644744#comment-16644744
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224006180
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolderTest.java
##
@@ -202,26 +250,38 @@ public void testGetAllFunctionsWithSignatures() {
public void testGetHoldersByFunctionNameWithVersion() {
List expectedResult = newJars.values().stream()
.flatMap(Collection::stream)
-.filter(f -> "lower".equals(f.getName()))
+.filter(f -> LOWER_FUNC_NAME.equals(f.getName()))
.map(FunctionHolder::getHolder)
.collect(Collectors.toList());
assertFalse(expectedResult.isEmpty());
AtomicInteger version = new AtomicInteger();
-compareTwoLists(expectedResult,
registryHolder.getHoldersByFunctionName("lower", version));
+compareTwoLists(expectedResult,
registryHolder.getHoldersByFunctionName(LOWER_FUNC_NAME, version));
assertEquals("Version number should match", version.get(),
registryHolder.getVersion());
}
@Test
public void testGetHoldersByFunctionName() {
-List expectedResult = newJars.values().stream()
-.flatMap(Collection::stream)
-.filter(f -> "lower".equals(f.getName()))
-.map(FunctionHolder::getHolder)
-.collect(Collectors.toList());
+List expectedUniqueResult = Lists.newArrayList();
Review comment:
Please use `new ArrayList<>` here and below.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644743#comment-16644743
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224001894
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
Review comment:
final?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644740#comment-16644740
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224002335
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
Review comment:
It's not save to cast: either change return type in `getFunctionRegistry()`
or add `instanceOf` check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644742#comment-16644742
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224005561
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java
##
@@ -161,6 +163,38 @@ public void removeJar(String jarName) {
}
}
+ /**
+ * Retrieves all functions (holders) associated with all the jars
+ * This is read operation, so several users can perform this operation at
the same time.
+ * @return list of all functions, mapped by their sources
+ */
+ public Map> getAllJarsWithFunctionHolders() {
+Map> allFunctionHoldersByJar = new
HashMap<>();
+
+try (@SuppressWarnings("unused") Closeable lock = readLock.open()) {
+ for (String jarName : jars.keySet()) {
+//Capture functionHolders here
+List drillFuncHolderList = new LinkedList<>();
Review comment:
Why LinkedList?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644746#comment-16644746
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r224002624
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
Review comment:
Why we need to wrap into ArrayList?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643976#comment-16643976 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-428318959 @arina-ielchiieva Rebased on top of Master which carries DRILL-6762's fix ( #1484 ), and removed reference to DRILL-6084 . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640774#comment-16640774 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-427588274 Made the suggested changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640773#comment-16640773
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r223187410
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ for (FunctionHolder dfhEntry : jarFunctionListMap.get(jarName)) {
+populateFunctionMap(functionMap, jarName, dfhEntry.getHolder());
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate the map for a given Jar-DrillFunctionHolder
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640772#comment-16640772
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r223187369
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
Review comment:
Put it in the comments
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee:
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639625#comment-16639625
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222959400
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
Review comment:
Not quite clear, please provide an example of what happens and example of
output as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Compo
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639624#comment-16639624
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222959209
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllJarsWithFunctionsHolders();
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ for (FunctionHolder dfhEntry : jarFunctionListMap.get(jarName)) {
+populateFunctionMap(functionMap, jarName, dfhEntry.getHolder());
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate the map for a given Jar-DrillFunctionHolder
Review comment:
I guess it's better to use java-doc to describe the method rather then
comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639626#comment-16639626
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222960733
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
##
@@ -291,6 +293,10 @@ public boolean isFunctionComplexOutput(String name) {
return false;
}
+ public LocalFunctionRegistry getLocalFunctionRegistry() {
Review comment:
But it's not used in this PR. You can getter in the PR for DRILL-6084.
Please remove the getter then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639623#comment-16639623
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222958985
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
Review comment:
1. I see, it's used in the constructor only then it's fine to leave it as is.
2. Java8 syntax?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639630#comment-16639630
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222960733
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
##
@@ -291,6 +293,10 @@ public boolean isFunctionComplexOutput(String name) {
return false;
}
+ public LocalFunctionRegistry getLocalFunctionRegistry() {
Review comment:
But it's not used in this PR. You can add getter in the PR for DRILL-6084.
Please remove the getter from this PR then.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638984#comment-16638984 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-427191976 @arina-ielchiieva updated the PR with an additional commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638862#comment-16638862 ] ASF GitHub Bot commented on DRILL-3988: --- kfaraaz commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-427165724 @kkhatua I checked there are no existing functional tests that do a DROP FUNCTION USING ... We need to add functional tests that drop functions which are defined in jar files and recreate and verify. /drillTEST_FRAMEWORK/drill-test-framework/framework/resources/Functional [test@dfc3012-45 Functional]# grep -ir "DROP FUNCTION USING JAR" * [test@dfc3012-45 Functional]# This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638773#comment-16638773
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222806883
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
Review comment:
Just realized.. that's a private method. Doesn't need a java-doc
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functio
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638627#comment-16638627
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222771518
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
Review comment:
The original loading of functions from `built-in` captures functions from
all the sources. When iterating over the non-`built-in` sources, the key helps
replace `FunctionInfo` instances that are wrongly shown as sourced from
`built-in`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
>
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638629#comment-16638629
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222771664
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
Review comment:
Ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to se
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638630#comment-16638630
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222771701
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
+ functionMap.put(funcSignatureKey, new FunctionInfo(name, signature,
returnType, jarName));
+}
+ }
+
+ @Override
+ public boolean hasNext() {
+return sortedIterator.hasNext();
+ }
+
+ @Override
+ public FunctionInfo next() {
+return sortedIterator.next();
+ }
+
+ public static class FunctionInfo {
Review comment:
Ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638625#comment-16638625
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222770729
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
Review comment:
Agreed. But the variable isn't used anywhere else in the `FunctionsIterator`
class, hence I wrote it in-place.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638623#comment-16638623
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222770403
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
Review comment:
That is because the UDFs are initialized and show up with the `source` as
`BUILT_IN`.
Iterating over holders from the non-`built-in` sources, I update the
`functionMap` with the true sources for those UDFs. I noticed this when a
function name was overloaded with a different signature.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638619#comment-16638619 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#discussion_r222769104 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java ## @@ -97,9 +98,13 @@ // function name, Map private final Map> functions; + // jar name, List + private final Map> jarFunctions; Review comment: The `functions` holder groups functionHolders from different sources as long as they share the same name. As a result, for a given function name, the source of a specific signature cannot be inferred. Since the holders non-primitive datatypes, they are not copied and only references maintained. So their additional footprint should be fairly small within the Drillbit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638581#comment-16638581
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222760646
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
##
@@ -291,6 +293,10 @@ public boolean isFunctionComplexOutput(String name) {
return false;
}
+ public LocalFunctionRegistry getLocalFunctionRegistry() {
Review comment:
For DRILL-6084 , the `WebServer` needs access to the registry to load the
list of functions for rendering in JavaScript.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638000#comment-16638000
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222602456
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
Review comment:
Please explain the purpose of this unique key.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacqu
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637999#comment-16637999
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222601123
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionImplementationRegistry.java
##
@@ -291,6 +293,10 @@ public boolean isFunctionComplexOutput(String name) {
return false;
}
+ public LocalFunctionRegistry getLocalFunctionRegistry() {
Review comment:
Why you need access to local function registry here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637996#comment-16637996 ] ASF GitHub Bot commented on DRILL-3988: --- arina-ielchiieva commented on a change in pull request #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#discussion_r222601521 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/registry/FunctionRegistryHolder.java ## @@ -97,9 +98,13 @@ // function name, Map private final Map> functions; + // jar name, List + private final Map> jarFunctions; Review comment: I don't see any value of adding one more holder to keep functions. Current two holders have enough information and its unique. Your holder just duplicates it. Please consider approach i have suggested before. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637995#comment-16637995
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222602309
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
Review comment:
You can make comparator class level variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637998#comment-16637998
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222602550
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
+ private void populateFunctionMap(Map functionMap,
String jarName, DrillFuncHolder dfh) {
+String registeredNames[] = dfh.getRegisteredNames();
+String signature = dfh.getInputParameters();
+String returnType = dfh.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ //Generate a uniqueKey to allow for fnName#fnSignature
+ String funcSignatureKey = new
StringBuilder(64).append(name).append('#').append(signature).toString();
+ functionMap.put(funcSignatureKey, new FunctionInfo(name, signature,
returnType, jarName));
+}
+ }
+
+ @Override
+ public boolean hasNext() {
+return sortedIterator.hasNext();
+ }
+
+ @Override
+ public FunctionInfo next() {
+return sortedIterator.next();
+ }
+
+ public static class FunctionInfo {
Review comment:
java-doc
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
> -
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637997#comment-16637997
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222602065
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
Review comment:
Why you have to populate built-in and jar functions separately?
`Map> jarFunctionListMap` contains
`LocalFunctionRegistry.BUILT_IN` for built-in functions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
> should be able to see (or run).
> - should we add a description annotation that could be included in the
> sys.functions table?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638001#comment-16638001
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222602489
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.FunctionImplementationRegistry;
+import org.apache.drill.exec.expr.fn.registry.FunctionHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.store.pojo.NonNullable;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ //private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+//Access Registry for function list
+FunctionImplementationRegistry funcImplRegistry =
(FunctionImplementationRegistry) context.getFunctionRegistry();
+Map> jarFunctionListMap =
funcImplRegistry.getAllFunctionsHoldersByJar();
+
+//Step 1: Load Built-In
+for (FunctionHolder builtInEntry :
jarFunctionListMap.get(LocalFunctionRegistry.BUILT_IN)) {
+ populateFunctionMap(functionMap, LocalFunctionRegistry.BUILT_IN,
builtInEntry.getHolder());
+}
+//Step 2: Load UDFs
+for (String jarName : new ArrayList<>(jarFunctionListMap.keySet())) {
+ if (!jarName.equals(LocalFunctionRegistry.BUILT_IN)) {
+for (FunctionHolder udfEntry : jarFunctionListMap.get(jarName)) {
+ populateFunctionMap(functionMap, jarName, udfEntry.getHolder());
+}
+ }
+}
+
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+});
+
+sortedIterator = functionList.iterator();
+ }
+
+ //Populate for give Jar-FunctionHolder Entry
Review comment:
Use java-doc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create a sys.functions table to expose available Drill functions
>
>
> Key: DRILL-3988
> URL: https://issues.apache.org/jira/browse/DRILL-3988
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Metadata
>Reporter: Jacques Nadeau
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Create a new sys.functions table that returns a list of all available
> functions.
> Key considerations:
> - one row per name or one per argument set. I'm inclined to latter so people
> can use queries to get to data.
> - we need to create a delineation between user functions and internal
> functions and only show user functions. 'CastInt' isn't something the user
>
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637809#comment-16637809 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-426881847 @kfaraaz these re unit tests, so we're not using the SQL interface to add or remove UDF jars. It's primarily testing the APIs. Existing functional tests should be covering that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637637#comment-16637637 ] ASF GitHub Bot commented on DRILL-3988: --- kfaraaz commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-426841861 Can you please add a unit test to drop a function using, DROP FUNCTION USING JAR '.jar'; and then query the sys.functions table and verify that the dropped function is not in the results returned, after it was dropped. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637607#comment-16637607 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua edited a comment on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-426834743 @arina-ielchiieva I've refactored as per suggestions and added tests. This has been rebased on top of current master, but does not have your PR #1484 as yet. I'm expecting that the batch committer will not encounter any merge conflicts. Could you please review these in the meanwhile? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637602#comment-16637602 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua commented on issue #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483#issuecomment-426834743 Refactored as per suggestions. Added tests. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637600#comment-16637600
]
ASF GitHub Bot commented on DRILL-3988:
---
kkhatua commented on a change in pull request #1483: DRILL-3988: Expose Drill
built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r222496506
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
Thanks, @arina-ielchiieva . Modified the PR as per your suggestion.
This is an automated message fro
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632558#comment-16632558
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r221377000
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
@kkhatua
To avoid this unpleasant parsing I would suggest you add method to
`FunctionRegistryHolder` which would return `ListMultimap`, where key would
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632553#comment-16632553
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r221377000
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
@kkhatua
To avoid this unpleasant parsing I would suggest you add method to
`FunctionRegistryHolder` which would return `ListMultimap`, where key would
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[
https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632537#comment-16632537
]
ASF GitHub Bot commented on DRILL-3988:
---
arina-ielchiieva commented on a change in pull request #1483: DRILL-3988:
Expose Drill built-in functions & UDFs in a system table
URL: https://github.com/apache/drill/pull/1483#discussion_r221377000
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/FunctionsIterator.java
##
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.registry.LocalFunctionRegistry;
+import org.apache.drill.exec.expr.fn.registry.RemoteFunctionRegistry;
+import org.apache.drill.exec.ops.ExecutorFragmentContext;
+import org.apache.drill.exec.proto.UserBitShared.Jar;
+import org.apache.drill.exec.proto.UserBitShared.Registry;
+import org.apache.drill.exec.store.pojo.NonNullable;
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+import org.apache.drill.shaded.guava.com.google.common.collect.ListMultimap;
+
+/**
+ * List functions as a System Table
+ */
+public class FunctionsIterator implements Iterator {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(FunctionsIterator.class);
+
+ private LocalFunctionRegistry localFunctionRegistry;
+ private RemoteFunctionRegistry remoteFunctionRegistry;
+ private Iterator sortedIterator;
+
+ public FunctionsIterator(ExecutorFragmentContext context) {
+Map functionMap = new HashMap();
+
+this.localFunctionRegistry = context.getLocalFunctionRegistry();
+listLocalFunctionRegistry(functionMap);
+
+this.remoteFunctionRegistry = context.getRemoteFunctionRegistry();
+listRemoteFunctionRegistry(functionMap);
+
+//Extract list & sort via comparator
+List functionList = new
ArrayList(functionMap.values());
+functionList.sort(initFunctionComparator());
+sortedIterator = functionList.iterator();
+ }
+
+ //Provide a function comparator
+ private Comparator initFunctionComparator() {
+//Initialize the comparator
+return new Comparator() {
+ @Override
+ public int compare(FunctionInfo o1, FunctionInfo o2) {
+int result = o1.name.compareTo(o2.name);
+if (result == 0) {
+ result = o1.signature.compareTo(o2.signature);
+}
+if (result == 0) {
+ return o1.returnType.compareTo(o2.returnType);
+}
+return result;
+ }
+};
+ }
+
+ //Loads local-registry functions into functionMap
+ private void listLocalFunctionRegistry(Map
functionMap) {
+ListMultimap builtInFuncHolderList =
localFunctionRegistry.getRegistryHolder().getAllFunctionsWithHolders();
+logger.debug("{} functions are currently initialized",
builtInFuncHolderList.size());
+
+Iterator builtInFuncIter =
builtInFuncHolderList.asMap().keySet().iterator();
+while (builtInFuncIter.hasNext()) {
+ String s = builtInFuncIter.next();
+ for (DrillFuncHolder drillFuncHolder : builtInFuncHolderList.get(s)) {
+String registeredNames[] = drillFuncHolder.getRegisteredNames();
+String signature = drillFuncHolder.getInputParameters();
+String returnType =
drillFuncHolder.getReturnType().getMinorType().toString();
+for (String name : registeredNames) {
+ FunctionInfo fnInfo = new FunctionInfo(name, signature, returnType,
LocalFunctionRegistry.BUILT_IN);
+ functionMap.put(generateKey(name, signature), fnInfo);
+}
+ }
+}
+ }
+
+ //Loads remote-registry functions into functionMap (updates if
+ private void listRemoteFunctionRegistry(Map
functionMap) {
Review comment:
@kkhatua
To avoid this unpleasant parsing I would suggest you add method to
`FunctionRegistryHolder` which would return `ListMultimap`, where key would
[jira] [Commented] (DRILL-3988) Create a sys.functions table to expose available Drill functions
[ https://issues.apache.org/jira/browse/DRILL-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632180#comment-16632180 ] ASF GitHub Bot commented on DRILL-3988: --- kkhatua opened a new pull request #1483: DRILL-3988: Expose Drill built-in functions & UDFs in a system table URL: https://github.com/apache/drill/pull/1483 This commit exposes available SQL functions in Drill and also detects UDFs that have been dynamically loaded into Drill. An example is shown below for 2 UDFs dynamically loaded into the cluster, along side the existing built-in functions that come with Drill. ``` 0: jdbc:drill:schema=sys> select source, count(*) as functionCount from sys.functions group by source; +-++ | source | functionCount | +-++ | built-in| 2704 | | simple-drill-function-1.0-SNAPSHOT.jar | 12 | | drill-url-tools-1.0.jar | 1 | +-++ 1 row selected (0.211 seconds) ``` The system table exposes information as shown. Since UDFs are lazily initialized (i.e. only when a SQL query needs it), the `returnType` is not available and listed as `n/a`. Once the UDF is initialized, the `returnType` is also available. The `random(FLOAT8-REQUIRED,FLOAT8-REQUIRED)` function is an example of a lazily loaded UDF (see `returnType`). The `url_parse(VARCHAR-REQUIRED)` function is an example of an initialized UDF (see `returnType`). Rest are built-in functions that meet the query's filter criteria. ``` 0: jdbc:drill:schema=sys>select * from sys.functions where name like 'random' or `name` like '%url%'; +-+--+-+-+ |name |signature | returnType | source | +-+--+-+-+ | parse_url | VARCHAR-REQUIRED | LATE| built-in | | random | | FLOAT8 | built-in | | random | FLOAT8-REQUIRED,FLOAT8-REQUIRED | n/a | simple-drill-function-1.0-SNAPSHOT.jar | | url_decode | VARCHAR-REQUIRED | VARCHAR | built-in | | url_encode | VARCHAR-REQUIRED | VARCHAR | built-in | | url_parse | VARCHAR-REQUIRED | LATE| drill-url-tools-1.0.jar | +-+--+-+-+ 6 rows selected (0.221 seconds) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create a sys.functions table to expose available Drill functions > > > Key: DRILL-3988 > URL: https://issues.apache.org/jira/browse/DRILL-3988 > Project: Apache Drill > Issue Type: Sub-task > Components: Metadata >Reporter: Jacques Nadeau >Assignee: Kunal Khatua >Priority: Major > Labels: newbie > Fix For: 1.15.0 > > > Create a new sys.functions table that returns a list of all available > functions. > Key considerations: > - one row per name or one per argument set. I'm inclined to latter so people > can use queries to get to data. > - we need to create a delineation between user functions and internal > functions and only show user functions. 'CastInt' isn't something the user > should be able to see (or run). > - should we add a description annotation that could be included in the > sys.functions table? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
