[
https://issues.apache.org/jira/browse/SPARK-11438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986390#comment-14986390
]
Xiangrui Meng commented on SPARK-11438:
---------------------------------------
[~piccolbo] `nondeterministic` means that Catalyst won't call `eval` multiple
times, for example,
{code}
df.select(randn(0L).as("r"))
.select(col("r") + 1.0, col("r") + 2.0)
{code}
won't trigger the random number generator more than once on a single record. It
doesn't mean that the result would be different in multiple job runs. You can
interpret `nondeterministic` as `do not evaluate more than once if it appears
multiple times in a single projection`. For UDFs, it is useful to have this
flag to avoid re-do heavy computations.
> Allow users to define nondeterministic UDFs
> -------------------------------------------
>
> Key: SPARK-11438
> URL: https://issues.apache.org/jira/browse/SPARK-11438
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Yin Huai
> Assignee: Yin Huai
>
> Right now, all UDFs are deterministic. It will be great if we allow users to
> define nondeterministic UDFs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
