[ https://issues.apache.org/jira/browse/SPARK-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706905#comment-15706905 ]
Zhan Zhang commented on SPARK-18637: ------------------------------------ Here is the comments from UDFType /** * If a UDF stores state based on the sequence of records it has processed, it * is stateful. A stateful UDF cannot be used in certain expressions such as * case statement and certain optimizations such as AND/OR short circuiting * don't apply for such UDFs, as they need to be invoked for each record. * row_sequence is an example of stateful UDF. A stateful UDF is considered to * be non-deterministic, irrespective of what deterministic() returns. * * @return true */ boolean stateful() default false; > Stateful UDF should be considered as nondeterministic > ----------------------------------------------------- > > Key: SPARK-18637 > URL: https://issues.apache.org/jira/browse/SPARK-18637 > Project: Spark > Issue Type: Bug > Reporter: Zhan Zhang > > If the annotation UDFType of a udf is stateful, it shoudl be considered as > non-deterministic. Otherwise, the catalyst may optimize the plan and return > the wrong result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org