[ https://issues.apache.org/jira/browse/SPARK-34804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305066#comment-17305066 ]
Sumeet Sharma commented on SPARK-34804: --------------------------------------- Proposed changes: [https://github.com/apache/spark/pull/31903] > registerFunction shouldnt be logging warning message for same function being > re-registered > ------------------------------------------------------------------------------------------ > > Key: SPARK-34804 > URL: https://issues.apache.org/jira/browse/SPARK-34804 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.1 > Reporter: Sumeet Sharma > Priority: Minor > > {code:java} > test("function registry warning") { > implicit val ss = spark > import ss.implicits._ > val dd = Seq((1),(2)).toDF("a") > val dd1 = udf((i: Int) => i*2) > (1 to 4).foreach { > _ => > dd.sparkSession.udf.register("function", dd1) > Thread.sleep(1000) > } > dd.withColumn("aa", expr("function(a)")).show(10) > }{code} > logs: > 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; > some spark core configurations may not take effect. > 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing SparkSession; > the static sql configurations will not take effect. > 21/03/19 22:39:39 WARN SparkSession$Builder : Using an existing > SparkSession; some spark core configurations may not take effect. > 21/03/19 22:39:43 WARN SimpleFunctionRegistry : The function function > replaced a previously registered function. > 21/03/19 22:39:44 WARN SimpleFunctionRegistry : The function function > replaced a previously registered function. > 21/03/19 22:39:45 WARN SimpleFunctionRegistry : The function function > replaced a previously registered function. > +----+--+ > |a|aa| > +----+--+ > |1|2| > |2|4| > +----+--+ > Basically in the FunctionRegistry implementation > {code:java} > override def registerFunction( > name: FunctionIdentifier, > info: ExpressionInfo, > builder: FunctionBuilder): Unit = synchronized { > val normalizedName = normalizeFuncName(name) > val newFunction = (info, builder) > functionBuilders.put(normalizedName, newFunction) match { > case Some(previousFunction) if previousFunction != newFunction => > logWarning(s"The function $normalizedName replaced a previously registered > function.") > case _ => > } > }{code} > The *previousFunction != newFunction* equality comparison is incorrect -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org