[jira] [Updated] (SPARK-23599) The UUID() expression is too non-deterministic
[
https://issues.apache.org/jira/browse/SPARK-23599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hovell updated SPARK-23599:
--
Fix Version/s: 2.3.1
> The UUID() expression is too non-deterministic
> --
>
> Key: SPARK-23599
> URL: https://issues.apache.org/jira/browse/SPARK-23599
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Herman van Hovell
>Assignee: Liang-Chi Hsieh
>Priority: Critical
> Fix For: 2.3.1, 2.4.0
>
>
> The current {{Uuid()}} expression uses {{java.util.UUID.randomUUID}} for UUID
> generation. There are a couple of major problems with this:
> - It is non-deterministic across task retries. This breaks Spark's processing
> model, and this will to very hard to trace bugs, like non-deterministic
> shuffles, duplicates and missing rows.
> - It uses a single secure random for UUID generation. This uses a single JVM
> wide lock, and this can lead to lock contention and other performance
> problems.
> We should move to something that is deterministic between retries. This can
> be done by using seeded PRNGs for which we set the seed during planning. It
> is important here to use a PRNG that provides enough entropy for creating a
> proper UUID.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23599) The UUID() expression is too non-deterministic
[
https://issues.apache.org/jira/browse/SPARK-23599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hovell updated SPARK-23599:
--
Priority: Critical (was: Major)
> The UUID() expression is too non-deterministic
> --
>
> Key: SPARK-23599
> URL: https://issues.apache.org/jira/browse/SPARK-23599
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Herman van Hovell
>Priority: Critical
>
> The current {{Uuid()}} expression uses {{java.util.UUID.randomUUID}} for UUID
> generation. There are a couple of major problems with this:
> - It is non-deterministic across task retries. This breaks Spark's processing
> model, and this will to very hard to trace bugs, like non-deterministic
> shuffles, duplicates and missing rows.
> - It uses a single secure random for UUID generation. This uses a single JVM
> wide lock, and this can lead to lock contention and other performance
> problems.
> We should move to something that is deterministic between retries. This can
> be done by using seeded PRNGs for which we set the seed during planning. It
> is important here to use a PRNG that provides enough entropy for creating a
> proper UUID.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
