[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682171#comment-16682171
]
ASF GitHub Bot commented on DRILL-4456:
---
ilooner closed pull request #1527: DRILL-4456: Add Hive translate UDF
URL: https://github.com/apache/drill/pull/1527
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
index cb00ede9bf0..0ec97785372 100644
---
a/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
+++
b/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
@@ -20,9 +20,12 @@
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
+import java.util.stream.Stream;
import org.apache.calcite.rel.type.RelDataType;
import org.apache.calcite.sql.SqlOperatorBinding;
+import org.apache.calcite.sql.fun.OracleSqlOperatorTable;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.calcite.sql.type.SqlReturnTypeInference;
import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.drill.common.config.DrillConfig;
@@ -38,6 +41,7 @@
import org.apache.drill.exec.planner.sql.HiveUDFOperator;
import org.apache.drill.exec.planner.sql.HiveUDFOperatorWithoutInference;
import org.apache.drill.exec.planner.sql.TypeInferenceUtils;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
import org.apache.hadoop.hive.ql.exec.Description;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.hive.ql.udf.UDFType;
@@ -48,8 +52,18 @@
import
org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap;
import org.apache.drill.shaded.guava.com.google.common.collect.Sets;
-public class HiveFunctionRegistry implements PluggableFunctionRegistry{
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+public class HiveFunctionRegistry implements PluggableFunctionRegistry {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+
+ /**
+ * Map for renaming UDFs. Keys of the map represent UDF names which should
be replaced
+ * and its values represent target UDF names.
+ */
+ private static final Map FUNCTION_REPLACE_MAP =
ImmutableMap. builder()
+ // renames Hive's TRANSLATE UDF to TRANSLATE3 due to CALCITE-1115
+ .put(SqlStdOperatorTable.TRANSLATE.getName().toLowerCase(),
+ OracleSqlOperatorTable.TRANSLATE3.getName().toLowerCase())
+ .build();
private ArrayListMultimap>
methodsGenericUDF = ArrayListMultimap.create();
private ArrayListMultimap> methodsUDF =
ArrayListMultimap.create();
@@ -102,27 +116,28 @@ public void register(DrillOperatorTable operatorTable) {
}
}
- private void register(Class clazz,
ArrayListMultimap> methods) {
+ private void register(Class clazz,
ArrayListMultimap> methods) {
Description desc = clazz.getAnnotation(Description.class);
-String[] names;
+Stream namesStream;
if (desc != null) {
- names = desc.name().split(",");
- for (int i=0; i name.replace('.', '_'));
}
+// Checks specified array of function names whether they should be replaced
+// using FUNCTION_REPLACE_MAP map.
+namesStream.map(String::toLowerCase)
+.map(functionName -> FUNCTION_REPLACE_MAP.getOrDefault(functionName,
functionName))
+.forEach(name -> methods.put(name, clazz));
+
UDFType type = clazz.getAnnotation(UDFType.class);
if (type != null && !type.deterministic()) {
nonDeterministicUDFs.add(clazz);
}
-
-
-for(int i=0; i Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ...
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681679#comment-16681679
]
ASF GitHub Bot commented on DRILL-4456:
---
vvysotskyi commented on a change in pull request #1527: DRILL-4456: Add Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r232316479
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -102,15 +116,15 @@ public void register(DrillOperatorTable operatorTable) {
}
}
- private void register(Class clazz,
ArrayListMultimap> methods) {
+ private void register(Class clazz,
ArrayListMultimap> methods) {
Review comment:
Thanks, done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681645#comment-16681645
]
ASF GitHub Bot commented on DRILL-4456:
---
ihuzenko commented on a change in pull request #1527: DRILL-4456: Add Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r232305628
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -102,15 +116,15 @@ public void register(DrillOperatorTable operatorTable) {
}
}
- private void register(Class clazz,
ArrayListMultimap> methods) {
+ private void register(Class clazz,
ArrayListMultimap> methods) {
Review comment:
If it's not required to have separate renameUDF(names) method, then it's
possible to simplify body of this method:
```
private void register(Class clazz,
ArrayListMultimap> methods) {
Description desc = clazz.getAnnotation(Description.class);
Stream names;
if (desc != null) {
names = Stream.of(desc.name().split(",")).map(String::trim);
} else {
names = Stream.of(clazz).map(Class::getName)
.map(name -> name.replace('.', '_'));
}
names.map(String::toLowerCase)
.map(funName -> FUNCTION_REPLACE_MAP.getOrDefault(funName, funName))
.forEach(udfFunName -> methods.put(udfFunName, clazz));
UDFType type = clazz.getAnnotation(UDFType.class);
if (type != null && !type.deterministic()) {
nonDeterministicUDFs.add(clazz);
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680007#comment-16680007
]
ASF GitHub Bot commented on DRILL-4456:
---
vvysotskyi commented on a change in pull request #1527: DRILL-4456: Fix Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231975877
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -48,8 +52,17 @@
import
org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap;
import org.apache.drill.shaded.guava.com.google.common.collect.Sets;
-public class HiveFunctionRegistry implements PluggableFunctionRegistry{
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+public class HiveFunctionRegistry implements PluggableFunctionRegistry {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+
+ /**
+ * Map for renaming Hive UDFs whose names satisfy the predicate in the key
by names from the map value.
+ */
+ private static final Map, String[]> FUNCTION_REPLACE_MAP
= ImmutableMap., String[]> builder()
Review comment:
Thanks, reworded a description and simplified the map to use target function
names as keys and predicates for checking as values.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680008#comment-16680008
]
ASF GitHub Bot commented on DRILL-4456:
---
vvysotskyi commented on a change in pull request #1527: DRILL-4456: Fix Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231976803
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -119,10 +132,26 @@ public void register(DrillOperatorTable operatorTable) {
nonDeterministicUDFs.add(clazz);
}
+names = renameUDF(names);
+
+for (String name : names) {
+ methods.put(name.toLowerCase(), clazz);
+}
+ }
-for(int i=0; i Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679802#comment-16679802
]
ASF GitHub Bot commented on DRILL-4456:
---
arina-ielchiieva commented on a change in pull request #1527: DRILL-4456: Fix
Hive translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231897328
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -119,10 +132,26 @@ public void register(DrillOperatorTable operatorTable) {
nonDeterministicUDFs.add(clazz);
}
+names = renameUDF(names);
+
+for (String name : names) {
+ methods.put(name.toLowerCase(), clazz);
+}
+ }
-for(int i=0; i Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679929#comment-16679929
]
ASF GitHub Bot commented on DRILL-4456:
---
vdiravka commented on a change in pull request #1527: DRILL-4456: Fix Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231862125
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConvertletTable.java
##
@@ -61,13 +69,36 @@
}
};
+ // Custom convertlet to avoid rewriting TIMESTAMP_DIFF by Calcite.
+ private static final SqlRexConvertlet TIMESTAMP_DIFF_CONVERTLET = (cx, call)
-> {
+SqlLiteral unitLiteral = call.operand(0);
+SqlIntervalQualifier qualifier =
+new SqlIntervalQualifier(unitLiteral.symbolValue(TimeUnit.class),
null, SqlParserPos.ZERO);
+
+List operands = Arrays.asList(
+cx.convertExpression(qualifier),
+cx.convertExpression(call.operand(1)),
+cx.convertExpression(call.operand(2)));
+
+RelDataTypeFactory typeFactory = cx.getTypeFactory();
+
+RelDataType returnType = typeFactory.createTypeWithNullability(
+typeFactory.createSqlType(SqlTypeName.BIGINT),
+cx.getValidator().getValidatedNodeType(call.operand(1)).isNullable()
+||
cx.getValidator().getValidatedNodeType(call.operand(2)).isNullable());
+
+return cx.getRexBuilder().makeCall(returnType,
+SqlStdOperatorTable.TIMESTAMP_DIFF, operands);
+ };
+
static {
// Use custom convertlet for EXTRACT function
map.put(SqlStdOperatorTable.EXTRACT, DrillExtractConvertlet.INSTANCE);
// SQRT needs it's own convertlet because calcite overrides it to POWER(x,
0.5)
// which is not suitable for Infinity value case
map.put(SqlStdOperatorTable.SQRT, SQRT_CONVERTLET);
map.put(SqlStdOperatorTable.COALESCE, COALESCE_CONVERTLET);
+map.put(SqlStdOperatorTable.TIMESTAMP_DIFF, TIMESTAMP_DIFF_CONVERTLET);
Review comment:
move the the string to the end of the block, since it is a last added
`SqlFunction`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679928#comment-16679928
]
ASF GitHub Bot commented on DRILL-4456:
---
vdiravka commented on a change in pull request #1527: DRILL-4456: Fix Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231943055
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -48,8 +52,17 @@
import
org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap;
import org.apache.drill.shaded.guava.com.google.common.collect.Sets;
-public class HiveFunctionRegistry implements PluggableFunctionRegistry{
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+public class HiveFunctionRegistry implements PluggableFunctionRegistry {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+
+ /**
+ * Map for renaming Hive UDFs whose names satisfy the predicate in the key
by names from the map value.
+ */
+ private static final Map, String[]> FUNCTION_REPLACE_MAP
= ImmutableMap., String[]> builder()
Review comment:
@vvysotskyi Try to simplify the logic for `FUNCTION_REPLACE_MAP`. Possibly
try to get rid from `String[]` (you can add all function names to this Map), so
in result it can be similar to
https://github.com/apache/drill/blob/master/logical/src/main/java/org/apache/drill/common/expression/fn/FunctionReplacementUtils.java#L42
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679803#comment-16679803
]
ASF GitHub Bot commented on DRILL-4456:
---
arina-ielchiieva commented on a change in pull request #1527: DRILL-4456: Fix
Hive translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231898546
##
File path:
contrib/storage-hive/core/src/main/java/org/apache/drill/exec/expr/fn/HiveFunctionRegistry.java
##
@@ -48,8 +52,17 @@
import
org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap;
import org.apache.drill.shaded.guava.com.google.common.collect.Sets;
-public class HiveFunctionRegistry implements PluggableFunctionRegistry{
- static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+public class HiveFunctionRegistry implements PluggableFunctionRegistry {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(HiveFunctionRegistry.class);
+
+ /**
+ * Map for renaming Hive UDFs whose names satisfy the predicate in the key
by names from the map value.
+ */
+ private static final Map, String[]> FUNCTION_REPLACE_MAP
= ImmutableMap., String[]> builder()
Review comment:
Please add better description.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679930#comment-16679930
]
ASF GitHub Bot commented on DRILL-4456:
---
vdiravka commented on a change in pull request #1527: DRILL-4456: Fix Hive
translate UDF
URL: https://github.com/apache/drill/pull/1527#discussion_r231862125
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/DrillConvertletTable.java
##
@@ -61,13 +69,36 @@
}
};
+ // Custom convertlet to avoid rewriting TIMESTAMP_DIFF by Calcite.
+ private static final SqlRexConvertlet TIMESTAMP_DIFF_CONVERTLET = (cx, call)
-> {
+SqlLiteral unitLiteral = call.operand(0);
+SqlIntervalQualifier qualifier =
+new SqlIntervalQualifier(unitLiteral.symbolValue(TimeUnit.class),
null, SqlParserPos.ZERO);
+
+List operands = Arrays.asList(
+cx.convertExpression(qualifier),
+cx.convertExpression(call.operand(1)),
+cx.convertExpression(call.operand(2)));
+
+RelDataTypeFactory typeFactory = cx.getTypeFactory();
+
+RelDataType returnType = typeFactory.createTypeWithNullability(
+typeFactory.createSqlType(SqlTypeName.BIGINT),
+cx.getValidator().getValidatedNodeType(call.operand(1)).isNullable()
+||
cx.getValidator().getValidatedNodeType(call.operand(2)).isNullable());
+
+return cx.getRexBuilder().makeCall(returnType,
+SqlStdOperatorTable.TIMESTAMP_DIFF, operands);
+ };
+
static {
// Use custom convertlet for EXTRACT function
map.put(SqlStdOperatorTable.EXTRACT, DrillExtractConvertlet.INSTANCE);
// SQRT needs it's own convertlet because calcite overrides it to POWER(x,
0.5)
// which is not suitable for Infinity value case
map.put(SqlStdOperatorTable.SQRT, SQRT_CONVERTLET);
map.put(SqlStdOperatorTable.COALESCE, COALESCE_CONVERTLET);
+map.put(SqlStdOperatorTable.TIMESTAMP_DIFF, TIMESTAMP_DIFF_CONVERTLET);
Review comment:
move the the string to the end of the block, since it is a last added
`SqlFunction`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679685#comment-16679685
]
ASF GitHub Bot commented on DRILL-4456:
---
vvysotskyi commented on issue #1527: DRILL-4456: Fix Hive translate UDF
URL: https://github.com/apache/drill/pull/1527#issuecomment-436978556
@vdiravka, done: https://github.com/apache/drill/pull/1528
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Fix For: 1.15.0
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319993#comment-16319993
]
Volodymyr Vysotskyi commented on DRILL-4456:
After rebasing to the Calcite1.15, this function may be used, but Drill throws
another exception:
{noformat}
Caused By (org.apache.calcite.sql.validate.SqlValidatorException) No match
found for function signature TRANSLATE3(, , )
{noformat}
This error appears because Calcite renames function name from {{translate}} to
{{translate3}}, but Hive has only function with name {{translate}}.
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
> Fix For: Future
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-4456) Hive translate function is not working
[
https://issues.apache.org/jira/browse/DRILL-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15268826#comment-15268826
]
Arina Ielchiieva commented on DRILL-4456:
-
Link to Calcite Jira - https://issues.apache.org/jira/browse/CALCITE-1115
> Hive translate function is not working
> --
>
> Key: DRILL-4456
> URL: https://issues.apache.org/jira/browse/DRILL-4456
> Project: Apache Drill
> Issue Type: Improvement
> Components: Functions - Hive
>Affects Versions: 1.5.0
>Reporter: Arina Ielchiieva
> Fix For: Future
>
>
> In Hive "select translate(name, 'A', 'B') from users" works fine.
> But in Drill "select translate(name, 'A', 'B') from hive.`users`" returns the
> following error:
> org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR:
> Encountered "," at line 1, column 22. Was expecting one of: "USING" ... "NOT"
> ... "IN" ... "BETWEEN" ... "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ...
> "<=" ... ">=" ... "<>" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ...
> "OR" ... "IS" ... "MEMBER" ... "SUBMULTISET" ... "MULTISET" ... "[" ... "."
> ... "(" ... while parsing SQL query: select translate(name, 'A', 'B') from
> hive.users ^ [Error Id: ba21956b-3285-4544-b3b2-fab68b95be1f on
> localhost:31010]
> Root cause:
> Calcite follows the standard SQL reference.
> SQL reference, ISO/IEC 9075-2:2011(E), section 6.30
> ::=
> TRANSLATE
> USING
> To fix:
> 1. add support to translate (expession, from_string, to_string) alternative
> syntax
> 2. add unit test in org.apache.drill.exec.fn.hive.TestInbuiltHiveUDFs
> Changes can be made directly in Calcite and then upgrade to appropriate
> Calcite version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
