[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902893#comment-16902893
]
ASF GitHub Bot commented on DRILL-7337:
---
vvysotskyi commented on pull request #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
> out.buffer = buffer = buffer.reallocIfNeeded(length);
> out.start = out.end = 0;
> for (VarCharHolder input : inputs) {
> for (int id = input.start; id < input.end; id++) {
> out.buffer.setByte(out.end++, input.buffer.getByte(id));
> }
> }
> }
> }
> {code}
> h2. Limitations connected with VarArg UDFs:
> * Specified nulls handling in FunctionTemplate does not affect vararg
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1682#comment-1682
]
ASF GitHub Bot commented on DRILL-7337:
---
arina-ielchiieva commented on issue #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#issuecomment-518187686
+1, please squash the commits.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
> out.buffer = buffer = buffer.reallocIfNeeded(length);
> out.start = out.end = 0;
> for (VarCharHolder input : inputs) {
> for (int id = input.start; id < input.end; id++) {
> out.buffer.setByte(out.end++, input.buffer.getByte(id));
> }
> }
> }
> }
> {code}
> h2. Limitations connected with VarArg UDFs:
> * Specified nulls handling in
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899940#comment-16899940
]
ASF GitHub Bot commented on DRILL-7337:
---
vvysotskyi commented on pull request #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#discussion_r310393284
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java
##
@@ -298,11 +368,19 @@ public int getParamCount() {
}
public boolean isConstant(int i) {
-return attributes.getParameters()[i].isConstant();
+return getAttributeParameter(i).isConstant();
+ }
+
+ public ValueReference getAttributeParameter(int i) {
Review comment:
Thanks, done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
>
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899939#comment-16899939
]
ASF GitHub Bot commented on DRILL-7337:
---
vvysotskyi commented on pull request #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#discussion_r310393344
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java
##
@@ -93,18 +96,30 @@
// TODO(Julien): verify there are a few of those and we can load them
Class fieldClass = field.getFieldClass();
if (param != null || output != null) {
+if (Object[].class.isAssignableFrom(fieldClass)) {
+ fieldClass = fieldClass.getComponentType();
+ varArgsCount++;
+} else if (varArgsCount > 0 && param != null) {
+ return failure("Vararg should be the last argument of function.",
func, field);
Review comment:
Thanks, done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899938#comment-16899938
]
ASF GitHub Bot commented on DRILL-7337:
---
vvysotskyi commented on pull request #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#discussion_r310392996
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java
##
@@ -262,6 +260,78 @@ protected void addProtectedBlock(ClassGenerator g,
JBlock sub, String body, H
}
}
+ /**
+ * Checks whether current function is var arg one and declares array for
storing vararg function arguments.
+ *
+ * @param model code model to generate the code
+ * @param jBlock block of code to be populated
+ * @param inputVariables array of input variables for current function
+ */
+ protected void declareVarArgArray(JCodeModel model, JBlock jBlock,
HoldingContainer[] inputVariables) {
+if (isVarArg()) {
Review comment:
Agree with you, thanks for pointing this, moved `isVarArg()` check outside
of the method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899632#comment-16899632
]
ASF GitHub Bot commented on DRILL-7337:
---
arina-ielchiieva commented on issue #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#issuecomment-517998861
Also could you please show how varargs functions will be described in
`sys.functions` table.
Example:
```
apache drill (sys)> select * from sys.functions limit 2;
+--+-++--+--+
| name |signature| returnType | source | internal |
+--+-++--+--+
| != | BIGINT-REQUIRED,BIGINT-REQUIRED | BIT| built-in | false|
| != | BIT-REQUIRED,BIT-REQUIRED | BIT| built-in | false|
+--+-++--+--+
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899628#comment-16899628
]
ASF GitHub Bot commented on DRILL-7337:
---
arina-ielchiieva commented on issue #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#issuecomment-517998861
Also could you please show how varargs functions will be described in
`sys.functions` table. Do we need new flag to filter them out?
Example:
```
apache drill (sys)> select * from sys.functions limit 2;
+--+-++--+--+
| name |signature| returnType | source | internal |
+--+-++--+--+
| != | BIGINT-REQUIRED,BIGINT-REQUIRED | BIT| built-in | false|
| != | BIT-REQUIRED,BIT-REQUIRED | BIT| built-in | false|
+--+-++--+--+
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899627#comment-16899627
]
ASF GitHub Bot commented on DRILL-7337:
---
arina-ielchiieva commented on pull request #1835: DRILL-7337: Add vararg UDFs
support
URL: https://github.com/apache/drill/pull/1835#discussion_r310384138
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java
##
@@ -93,18 +96,30 @@
// TODO(Julien): verify there are a few of those and we can load them
Class fieldClass = field.getFieldClass();
if (param != null || output != null) {
+if (Object[].class.isAssignableFrom(fieldClass)) {
+ fieldClass = fieldClass.getComponentType();
+ varArgsCount++;
+} else if (varArgsCount > 0 && param != null) {
+ return failure("Vararg should be the last argument of function.",
func, field);
Review comment:
```suggestion
return failure("Vararg should be the last argument in the
function.", func, field);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
>
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899626#comment-16899626
]
ASF GitHub Bot commented on DRILL-7337:
---
arina-ielchiieva commented on pull request #1835: DRILL-7337: Add vararg UDFs
support
URL: https://github.com/apache/drill/pull/1835#discussion_r310384111
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java
##
@@ -298,11 +368,19 @@ public int getParamCount() {
}
public boolean isConstant(int i) {
-return attributes.getParameters()[i].isConstant();
+return getAttributeParameter(i).isConstant();
+ }
+
+ public ValueReference getAttributeParameter(int i) {
Review comment:
Please add java doc thus there will be no need in the comment in the code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end -
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899625#comment-16899625
]
ASF GitHub Bot commented on DRILL-7337:
---
arina-ielchiieva commented on pull request #1835: DRILL-7337: Add vararg UDFs
support
URL: https://github.com/apache/drill/pull/1835#discussion_r310384088
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java
##
@@ -262,6 +260,78 @@ protected void addProtectedBlock(ClassGenerator g,
JBlock sub, String body, H
}
}
+ /**
+ * Checks whether current function is var arg one and declares array for
storing vararg function arguments.
+ *
+ * @param model code model to generate the code
+ * @param jBlock block of code to be populated
+ * @param inputVariables array of input variables for current function
+ */
+ protected void declareVarArgArray(JCodeModel model, JBlock jBlock,
HoldingContainer[] inputVariables) {
+if (isVarArg()) {
Review comment:
Not sure if this check should be done here, I would assume its caller
responsibility first check `isVarArg()` and then call `declareVarArgArray`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
>
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899538#comment-16899538
]
ASF GitHub Bot commented on DRILL-7337:
---
cgivre commented on issue #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835#issuecomment-517966762
This is a really useful feature!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
> out.buffer = buffer = buffer.reallocIfNeeded(length);
> out.start = out.end = 0;
> for (VarCharHolder input : inputs) {
> for (int id = input.start; id < input.end; id++) {
> out.buffer.setByte(out.end++, input.buffer.getByte(id));
> }
> }
> }
> }
> {code}
> h2. Limitations connected with VarArg UDFs:
> * Specified nulls handling in FunctionTemplate does not affect vararg
>
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899450#comment-16899450
]
ASF GitHub Bot commented on DRILL-7337:
---
vvysotskyi commented on pull request #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835
For details please see
[DRILL-7337](https://issues.apache.org/jira/browse/DRILL-7337).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
> out.buffer = buffer = buffer.reallocIfNeeded(length);
> out.start = out.end = 0;
> for (VarCharHolder input : inputs) {
> for (int id = input.start; id < input.end; id++) {
> out.buffer.setByte(out.end++, input.buffer.getByte(id));
> }
> }
> }
> }
> {code}
> h2. Limitations connected with VarArg UDFs:
> * Specified nulls handling in
