[ https://issues.apache.org/jira/browse/SPARK-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706961#comment-15706961 ]
Zhan Zhang commented on SPARK-18637: ------------------------------------ [~hvanhovell] It is an annotation. /** * UDFType annotations are used to describe properties of a UDF. This gives * important information to the optimizer. * If the UDF is not deterministic, or if it is stateful, it is necessary to * annotate it as such for correctness. * */ @Public @Evolving @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) @Inherited public @interface UDFType { /** * Certain optimizations should not be applied if UDF is not deterministic. * Deterministic UDF returns same result each time it is invoked with a * particular input. This determinism just needs to hold within the context of * a query. * * @return true if the UDF is deterministic */ boolean deterministic() default true; /** * If a UDF stores state based on the sequence of records it has processed, it * is stateful. A stateful UDF cannot be used in certain expressions such as * case statement and certain optimizations such as AND/OR short circuiting * don't apply for such UDFs, as they need to be invoked for each record. * row_sequence is an example of stateful UDF. A stateful UDF is considered to * be non-deterministic, irrespective of what deterministic() returns. * * @return true */ boolean stateful() default false; /** * A UDF is considered distinctLike if the UDF can be evaluated on just the * distinct values of a column. Examples include min and max UDFs. This * information is used by metadata-only optimizer. * * @return true if UDF is distinctLike */ boolean distinctLike() default false; /** * Using in analytical functions to specify that UDF implies an ordering * * @return true if the function implies order */ boolean impliesOrder() default false; } > Stateful UDF should be considered as nondeterministic > ----------------------------------------------------- > > Key: SPARK-18637 > URL: https://issues.apache.org/jira/browse/SPARK-18637 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Zhan Zhang > > If the annotation UDFType of a udf is stateful, it shoudl be considered as > non-deterministic. Otherwise, the catalyst may optimize the plan and return > the wrong result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org