What's the result of "list jar" in both 1.3.1 and 1.4.0, please check if there
is any difference
At 2015-07-15 08:10:44, "ogoh" <oke...@gmail.com> wrote:
>Hello,
>I am using SparkSQL along with ThriftServer so that we can access using Hive
>queries.
>With Spark 1.3.1, I can register UDF function. But, Spark 1.4.0 doesn't work
>for that. The jar of the udf is same.
>Below is logs:
>I appreciate any advice.
>
>
>== With Spark 1.4
>Beeline version 1.4.0 by Apache Hive
>
>0: jdbc:hive2://localhost:10000> add jar
>hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar;
>
>0: jdbc:hive2://localhost:10000> create temporary function parse_trace as
>'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation';
>
>15/07/14 23:49:43 DEBUG transport.TSaslTransport: writing data length: 206
>
>15/07/14 23:49:43 DEBUG transport.TSaslTransport: CLIENT: reading data
>length: 201
>
>Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED:
>Execution Error, return code 1 from
>org.apache.hadoop.hive.ql.exec.FunctionTask (state=,code=0)
>
>
>== With Spark 1.3.1:
>
>Beeline version 1.3.1 by Apache Hive
>
>0: jdbc:hive2://localhost:10001> add jar
>hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar;
>
>+---------+
>
>| Result |
>
>+---------+
>
>+---------+
>
>No rows selected (1.313 seconds)
>
>0: jdbc:hive2://localhost:10001> create temporary function parse_trace as
>'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation';
>
>+---------+
>
>| result |
>
>+---------+
>
>+---------+
>
>No rows selected (0.999 seconds)
>
>
>=== The logs of ThriftServer of Spark 1.4.0
>
>15/07/14 23:49:43 INFO SparkExecuteStatementOperation: Running query 'create
>temporary function parse_trace as
>'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation''
>
>15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary
>function parse_trace as
>'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'
>
>15/07/14 23:49:43 INFO ParseDriver: Parse Completed
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.run
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=TimeToSubmit
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO Driver: Concurrency mode is disabled, not creating a
>lock manager
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=compile
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=parse
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary
>function parse_trace as
>'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'
>
>15/07/14 23:49:43 INFO ParseDriver: Parse Completed
>
>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=parse
>start=1436917783106 end=1436917783106 duration=0
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=semanticAnalyze
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO HiveMetaStore: 2: get_database: default
>
>15/07/14 23:49:43 INFO audit: ugi=anonymous ip=unknown-ip-addr
>cmd=get_database: default
>
>15/07/14 23:49:43 INFO HiveMetaStore: 2: Opening raw store with
>implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>
>15/07/14 23:49:43 INFO ObjectStore: ObjectStore, initialize called
>
>15/07/14 23:49:43 INFO MetaStoreDirectSql: MySQL check failed, assuming we
>are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64),
>after : "".
>
>15/07/14 23:49:43 INFO Query: Reading in results for query
>"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
>closing
>
>15/07/14 23:49:43 INFO ObjectStore: Initialized ObjectStore
>
>15/07/14 23:49:43 INFO FunctionSemanticAnalyzer: analyze done
>
>15/07/14 23:49:43 INFO Driver: Semantic Analysis Completed
>
>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=semanticAnalyze
>start=1436917783106 end=1436917783114 duration=8
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO Driver: Returning Hive schema:
>Schema(fieldSchemas:null, properties:null)
>
>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=compile
>start=1436917783106 end=1436917783114 duration=8
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.execute
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO Driver: Starting command: create temporary function
>parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'
>
>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=TimeToSubmit
>start=1436917783105 end=1436917783115 duration=10
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=runTasks
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=task.FUNCTION.Stage-0
>from=org.apache.hadoop.hive.ql.Driver>
>
>15/07/14 23:49:43 ERROR Task: FAILED: Class
>com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation not found
>
>15/07/14 23:49:43 INFO FunctionTask: create function:
>java.lang.ClassNotFoundException:
>com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>
>
>
>--
>View this message in context:
>http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-4-can-t-accept-registration-of-UDF-tp23840.html
>Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>For additional commands, e-mail: user-h...@spark.apache.org
>
