[ https://issues.apache.org/jira/browse/SPARK-24544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546305#comment-16546305 ]
Apache Spark commented on SPARK-24544: -------------------------------------- User 'caneGuy' has created a pull request for this issue: https://github.com/apache/spark/pull/21790 > Print actual failure cause when look up function failed > ------------------------------------------------------- > > Key: SPARK-24544 > URL: https://issues.apache.org/jira/browse/SPARK-24544 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.1.0, 2.3.1 > Reporter: zhoukang > Priority: Minor > > When we operate as below: > {code} > 0: jdbc:hive2://xxx/> create function funnel_analysis as > 'com.xxx.hive.extend.udf.UapFunnelAnalysis'; > 0: jdbc:hive2://xxx/> use mifi; > +---------+--+ > | Result | > +---------+--+ > +---------+--+ > 0: jdbc:hive2://xxx/> select funnel_analysis(1,",",1,''); > Error: org.apache.spark.sql.AnalysisException: Undefined function: > 'funnel_analysis'. This function is neither a registered temporary function > nor a permanent function registered in the database 'xxx'.; line 1 pos 7 > (state=,code=0) > 0: jdbc:hive2://xxx/> describe function funnel_analysis; > +-----------------------------------------------------------+--+ > | function_desc | > +-----------------------------------------------------------+--+ > | Function: mifi.funnel_analysis | > | Class: com.xxx.hive.extend.udf.UapFunnelAnalysis | > | Usage: N/A. | > +-----------------------------------------------------------+--+ > {code} > We can see describe funtion will get right information,but when we actually > use this funtion,we will get an undefined exception. > Which is really misleading,the real cause is below: > {code} > 2018-06-13,17:15:16,725 INFO > org.apache.spark.sql.hive.KuduHiveSessionCatalog: Error: > org.apache.spark.sql.AnalysisException: No handler for Hive UDF > 'com.xiaomi.mifi.hive.extend.udf.UapFunnelAnalysis': > java.lang.IllegalStateException: Should not be called directly; > at > org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.initialize(GenericUDTF.java:72) > at > org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector$lzycompute(hiveUDFs.scala:204) > at > org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector(hiveUDFs.scala:204) > at > org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema$lzycompute(hiveUDFs.scala:212) > at > org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema(hiveUDFs.scala:212) > at > org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:146) > at > org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionBuilder$1.apply(HiveSessionCatalog.scala:122) > at > org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:94) > at > org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1060) > at > org.apache.spark.sql.hive.HiveSessionCatalog.org$apache$spark$sql$hive$HiveSessionCatalog$$super$lookupFunction(HiveSessionCatalog.scala:192) > at > org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$6.apply(HiveSessionCatalog.scala:192) > at > org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$6.apply(HiveSessionCatalog.scala:192) > at scala.util.Try$.apply(Try.scala:192) > at > org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction0(HiveSessionCatalog.scala:192) > at > org.apache.spark.sql.hive.HiveSessionCatalog.lookupFunction(HiveSessionCatalog.scala:177) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6$$anonfun$applyOrElse$39.apply(Analyzer.scala:900) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6$$anonfun$applyOrElse$39.apply(Analyzer.scala:900) > at > org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6.applyOrElse(Analyzer.scala:899) > at > org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$13$$anonfun$applyOrElse$6.applyOrElse(Analyzer.scala:887) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:268) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:268) > at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70) > at > org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:267) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:273) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:273) > {code} > I think we should print this actual error for quick debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org