I want to define some UDFs in my spark ENV. And server it in thrift server. So I can use these UDFs in my beeline connection.
At first I tried start it with udf-jars and create functions in hive. In spark-sql , I can add temp functions like "CREATE TEMPORARY FUNCTION bsdUpper AS 'org.hue.udf.MyUpper';" , and it works well , but when I add functions like "CREATE FUNCTION bsdupperlong AS 'org.hue.udf.MyUpper' USING JAR 'hdfs://ns1/bsdtest2/myudfs.jar';" . this command goes well , and I can see this function in metastore. But when I use this function "bsdupperlong" as I expected, it turns out , spark cannot find this function. When I add this function again , it show an exception , means this function already exist. Then I found this unresolved ISSU : https://issues.apache.org/jira/browse/SPARK-11609 So, .. It's a bug that has not been fixed. Then I found this. https://mail-archives.apache.org/mod_mbox/spark-user/201510.mbox/%3CCADONuiR elyy7ebcj6qr3guo023upmvnt_d_gadrnbavhpor...@mail.gmail.com%3E It tells me that I can Register in context , then start the Thrift Server using startWithContext from the spark shell. But I failed on google to found an article to show me how start the Thrift Server using startWithContext from the spark shell is done. Is anybody can help me with that ? Or find some other solution , that I can use UDFs in my beeline client. Thanks a lot. Version of spark . spark-1.5.2 Hadoop 2.7.1 Hive 1.2.1 UDF code in jar is like this . package org.hue.udf; import org.apache.hadoop.hive.ql.exec.UDF; import org.apache.hadoop.io.Text; import java.util.UUID; public final class MyUUID extends UDF { public String evaluate() { UUID uuid = UUID.randomUUID(); return uuid.toString(); } }
