[jira] [Assigned] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng reassigned SPARK-14393:
-
Assignee: Xiangrui Meng
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0, 2.0.0, 2.0.1
>Reporter: Jason Piper
>Assignee: Xiangrui Meng
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-14393:
Assignee: Xiangrui Meng (was: Apache Spark)
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0, 2.0.0, 2.0.1
>Reporter: Jason Piper
>Assignee: Xiangrui Meng
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-14393:
Assignee: Apache Spark (was: Xiangrui Meng)
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0, 2.0.0, 2.0.1
>Reporter: Jason Piper
>Assignee: Apache Spark
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
