[jira] [Commented] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely
[
https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646760#comment-16646760
]
Apache Spark commented on SPARK-25690:
--
User 'maryannxue' has created a pull request for this issue:
https://github.com/apache/spark/pull/22701
> Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied
> infinitely
> ---
>
> Key: SPARK-25690
> URL: https://issues.apache.org/jira/browse/SPARK-25690
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 2.4.0
>Reporter: Maryann Xue
>Priority: Major
>
> This was fixed in SPARK-24891 and was then broken by SPARK-25044.
> The unit test in {{AnalysisSuite}} added in SPARK-24891 should have failed
> but didn't because it wasn't properly updated after the {{ScalaUDF}}
> constructor signature change. In the meantime, the other two end-to-end tests
> added in SPARK-24891 were shadowed by SPARK-24865.
> So the unit test mentioned above, if updated properly, can reproduce this
> issue:
> {code:java}
> test("SPARK-24891 Fix HandleNullInputsForUDF rule") {
> val a = testRelation.output(0)
> val func = (x: Int, y: Int) => x + y
> val udf1 = ScalaUDF(func, IntegerType, a :: a :: Nil, nullableTypes = false
> :: false :: Nil)
> val udf2 = ScalaUDF(func, IntegerType, a :: udf1 :: Nil, nullableTypes =
> false :: false :: Nil)
> val plan = Project(Alias(udf2, "")() :: Nil, testRelation)
> comparePlans(plan.analyze, plan.analyze.analyze)
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644273#comment-16644273 ] Xiao Li commented on SPARK-25690: - The changes made in https://issues.apache.org/jira/browse/SPARK-25044 broke the rule HandleNullInputsForUDF. It is not idempotent any more. Since AnalysisBarrier is removed in 2.4 release, this is not blocker based on my evaluation. However, we should still fix it. cc [~cloud_fan] [~rxin] > Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied > infinitely > --- > > Key: SPARK-25690 > URL: https://issues.apache.org/jira/browse/SPARK-25690 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 2.4.0 >Reporter: Maryann Xue >Priority: Major > > This was fixed in SPARK-24891 and was then broken by SPARK-25044. > The tests added in SPARK-24891 were not good enough and the expected failures > were shadowed by SPARK-24865. For more details, please refer to SPARK-25650. > Code changes and tests in > [https://github.com/apache/spark/pull/22060/files#diff-f70523b948b7af21abddfa3ab7e1d7d6R72] > can help reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25690) Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied infinitely
[ https://issues.apache.org/jira/browse/SPARK-25690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644117#comment-16644117 ] Sean Owen commented on SPARK-25690: --- I'm unclear what the particular problem is here. Is there a test case we can add that demonstrates the issue? Does it only affect 2.12? Why are there two separate child JIRAs? This is not fixed in 2.4, according to you, nor should I be assigned. > Analyzer rule "HandleNullInputsForUDF" does not stabilize and can be applied > infinitely > --- > > Key: SPARK-25690 > URL: https://issues.apache.org/jira/browse/SPARK-25690 > Project: Spark > Issue Type: Sub-task > Components: Spark Core, SQL >Affects Versions: 2.4.0 >Reporter: Maryann Xue >Assignee: Sean Owen >Priority: Major > > This was fixed in SPARK-24891 and was then broken by SPARK-25044. > The tests added in SPARK-24891 were not good enough and the expected failures > were shadowed by SPARK-24865. For more details, please refer to SPARK-25650. > Code changes and tests in > [https://github.com/apache/spark/pull/22060/files#diff-f70523b948b7af21abddfa3ab7e1d7d6R72] > can help reproduce the issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
