Hello, I am using SparkSQL along with ThriftServer so that we can access using Hive queries. With Spark 1.3.1, I can register UDF function. But, Spark 1.4.0 doesn't work for that. The jar of the udf is same. Below is logs: I appreciate any advice.
== With Spark 1.4 Beeline version 1.4.0 by Apache Hive 0: jdbc:hive2://localhost:10000> add jar hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar; 0: jdbc:hive2://localhost:10000> create temporary function parse_trace as 'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation'; 15/07/14 23:49:43 DEBUG transport.TSaslTransport: writing data length: 206 15/07/14 23:49:43 DEBUG transport.TSaslTransport: CLIENT: reading data length: 201 Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=,code=0) == With Spark 1.3.1: Beeline version 1.3.1 by Apache Hive 0: jdbc:hive2://localhost:10001> add jar hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar; +---------+ | Result | +---------+ +---------+ No rows selected (1.313 seconds) 0: jdbc:hive2://localhost:10001> create temporary function parse_trace as 'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation'; +---------+ | result | +---------+ +---------+ No rows selected (0.999 seconds) === The logs of ThriftServer of Spark 1.4.0 15/07/14 23:49:43 INFO SparkExecuteStatementOperation: Running query 'create temporary function parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'' 15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary function parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation' 15/07/14 23:49:43 INFO ParseDriver: Parse Completed 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO Driver: Concurrency mode is disabled, not creating a lock manager 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary function parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation' 15/07/14 23:49:43 INFO ParseDriver: Parse Completed 15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=parse start=1436917783106 end=1436917783106 duration=0 from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO HiveMetaStore: 2: get_database: default 15/07/14 23:49:43 INFO audit: ugi=anonymous ip=unknown-ip-addr cmd=get_database: default 15/07/14 23:49:43 INFO HiveMetaStore: 2: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 15/07/14 23:49:43 INFO ObjectStore: ObjectStore, initialize called 15/07/14 23:49:43 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "". 15/07/14 23:49:43 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing 15/07/14 23:49:43 INFO ObjectStore: Initialized ObjectStore 15/07/14 23:49:43 INFO FunctionSemanticAnalyzer: analyze done 15/07/14 23:49:43 INFO Driver: Semantic Analysis Completed 15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=semanticAnalyze start=1436917783106 end=1436917783114 duration=8 from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null) 15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=compile start=1436917783106 end=1436917783114 duration=8 from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO Driver: Starting command: create temporary function parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation' 15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=TimeToSubmit start=1436917783105 end=1436917783115 duration=10 from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=task.FUNCTION.Stage-0 from=org.apache.hadoop.hive.ql.Driver> 15/07/14 23:49:43 ERROR Task: FAILED: Class com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation not found 15/07/14 23:49:43 INFO FunctionTask: create function: java.lang.ClassNotFoundException: com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation at java.net.URLClassLoader$1.run(URLClassLoader.java:372) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-4-can-t-accept-registration-of-UDF-tp23840.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org