[GitHub] spark issue #23153: [SPARK-26147][SQL] only pull out unevaluable python udf ...

[xuanyuanking]

Github user xuanyuanking commented on the issue:

    https://github.com/apache/spark/pull/23153
  
    Thanks for the fix from Wenchen, 
    ```
    the suites should also construct the dummy python udf from both side.
    ```
    I fix the suite locally, they can be simply modified like:
    ```
    diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
    index d3867f2b6b..a0f8ae2fc7 100644
    --- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
    +++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PullOutPythonUDFInJoinConditionSuite.scala
    @@ -40,13 +40,18 @@ class PullOutPythonUDFInJoinConditionSuite extends 
PlanTest {
             CheckCartesianProducts) :: Nil
       }
    
    -  val testRelationLeft = LocalRelation('a.int, 'b.int)
    -  val testRelationRight = LocalRelation('c.int, 'd.int)
    +  val attrA = 'a.int
    +  val attrB = 'b.int
    +  val attrC = 'c.int
    +  val attrD = 'd.int
    +
    +  val testRelationLeft = LocalRelation(attrA, attrB)
    +  val testRelationRight = LocalRelation(attrC, attrD)
    
       // Dummy python UDF for testing. Unable to execute.
       val pythonUDF = PythonUDF("pythonUDF", null,
         BooleanType,
    -    Seq.empty,
    +    Seq(attrA, attrC),
         PythonEvalType.SQL_BATCHED_UDF,
         udfDeterministic = true)
    
    @@ -118,7 +123,7 @@ class PullOutPythonUDFInJoinConditionSuite extends 
PlanTest {
       test("pull out whole complex condition with multiple python udf") {
         val pythonUDF1 = PythonUDF("pythonUDF1", null,
           BooleanType,
    -      Seq.empty,
    +      Seq(attrA, attrC),
           PythonEvalType.SQL_BATCHED_UDF,
           udfDeterministic = true)
         val condition = (pythonUDF || 'a.attr === 'c.attr) && pythonUDF1
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org