[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784134#comment-16784134 ]
Chakravarthi edited comment on SPARK-26602 at 3/5/19 1:52 PM: -------------------------------------------------------------- Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue is,Insert into table fails after querying the UDF which is loaded with wrong hdfs path. Below are the steps to reproduce this issue: 1) create a table. sql("create table table1(I int)"); 2) create udf using invalid hdfs path. sql("CREATE FUNCTION before_fix AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR 'hdfs:///tmp/notexist.jar'") 3) Do select on the UDF and you will get exception as "Failed to read external resource". sql(" select before_fix('2018-03-09')"). 4) perform insert table. sql("insert into table1 values(1)").show Here ,insert should work.but is fails. was (Author: chakravarthi): Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue is,Insert into table fails after querying the UDF which is loaded with wrong hdfs path. Below are the steps to reproduce this issue: 1) create a table. sql("create table check_udf(I int)"); 2) create udf using invalid hdfs path. sql("CREATE FUNCTION before_fix AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR 'hdfs:///tmp/notexist.jar'") 3) Do select on the UDF and you will get exception as "Failed to read external resource". sql(" select before_fix('2018-03-09')"). 4) perform insert table. sql("insert into check_udf values(1)").show Here ,insert should work.but is fails. > Insert into table fails after querying the UDF which is loaded with wrong > hdfs path > ----------------------------------------------------------------------------------- > > Key: SPARK-26602 > URL: https://issues.apache.org/jira/browse/SPARK-26602 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.0 > Reporter: Haripriya > Priority: Major > Attachments: beforeFixUdf.txt > > > In sql, > 1.Query the existing udf(say myFunc1) > 2. create and select the udf registered with incorrect path (say myFunc2) > 3.Now again query the existing udf in the same session - Wil throw exception > stating that couldn't read resource of myFunc2's path > 4.Even the basic operations like insert and select will fail giving the same > error > Result: > java.lang.RuntimeException: Failed to read external resource > hdfs:///tmp/hari_notexists1/two_udfs.jar > at > org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288) > at > org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163) > at > org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149) > at > org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275) > at > org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213) > at > org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212) > at > org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706) > at > org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696) > at > org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841) > at > org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org