[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-14393:
Priority: Blocker (was: Major)
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0, 2.0.0, 2.0.1
>Reporter: Jason Piper
>Assignee: Xiangrui Meng
>Priority: Blocker
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-14393:
Target Version/s: 2.1.0
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0, 2.0.0, 2.0.1
>Reporter: Jason Piper
>Assignee: Xiangrui Meng
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-14393:
-
Affects Version/s: 2.0.0
2.0.1
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0, 2.0.0, 2.0.1
>Reporter: Jason Piper
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng updated SPARK-14393:
--
Labels: correctness (was: )
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Jason Piper
> Labels: correctness
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> | 0|
> +---+
> >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> | 1|
> | 0|
> | 0|
> | 1|
> | 2|
> | 3|
> | 0|
> | 1|
> | 2|
> +---+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Piper updated SPARK-14393:
Description:
When utilising monotonicallyIncreasingId with a coalesce, it appears that every
partition uses the same offset (0) leading to non-monotonically increasing IDs.
See examples below
{code}
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
+---+
|monotonicallyincreasingid()|
+---+
|25769803776|
|51539607552|
|77309411328|
| 103079215104|
| 128849018880|
| 163208757248|
| 188978561024|
| 214748364800|
| 240518168576|
| 266287972352|
+---+
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
+---+
>>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 1|
| 0|
| 0|
| 1|
| 2|
| 3|
| 0|
| 1|
| 2|
+---+
{code}
was:
When utilising monotonicallyIncreasingId with a coalesce, it appears that every
partition uses the same offset (0) leading to non-monotonically increasing IDs.
See examples below
{code}
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
+---+
|monotonicallyincreasingid()|
+---+
|25769803776|
|51539607552|
|77309411328|
| 103079215104|
| 128849018880|
| 163208757248|
| 188978561024|
| 214748364800|
| 240518168576|
| 266287972352|
+---+
```
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
+---+
>>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 1|
| 0|
| 0|
| 1|
| 2|
| 3|
| 0|
| 1|
| 2|
+---+
{code}
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Jason Piper
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> |
[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[ https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Piper updated SPARK-14393: Description: When utilising monotonicallyIncreasingId with a coalesce, it appears that every partition uses the same offset (0) leading to non-monotonically increasing IDs. See examples below ``` >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show() +---+ |monotonicallyincreasingid()| +---+ |25769803776| |51539607552| |77309411328| | 103079215104| | 128849018880| | 163208757248| | 188978561024| | 214748364800| | 240518168576| | 266287972352| +---+ ``` >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show() +---+ |monotonicallyincreasingid()| +---+ | 0| | 0| | 0| | 0| | 0| | 0| | 0| | 0| | 0| | 0| +---+ >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show() +---+ |monotonicallyincreasingid()| +---+ | 0| | 1| | 0| | 0| | 1| | 2| | 3| | 0| | 1| | 2| +---+ was: When utilising monotonicallyIncreasingId with a coalesce, it appears that every partition uses the same offset (0) leading to non-monotonically increasing IDs. See examples below >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show() +---+ |monotonicallyincreasingid()| +---+ |25769803776| |51539607552| |77309411328| | 103079215104| | 128849018880| | 163208757248| | 188978561024| | 214748364800| | 240518168576| | 266287972352| +---+ >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show() +---+ |monotonicallyincreasingid()| +---+ | 0| | 0| | 0| | 0| | 0| | 0| | 0| | 0| | 0| | 0| +---+ >>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show() +---+ |monotonicallyincreasingid()| +---+ | 0| | 1| | 0| | 0| | 1| | 2| | 3| | 0| | 1| | 2| +---+ > monotonicallyIncreasingId not monotonically increasing with downstream > coalesce > --- > > Key: SPARK-14393 > URL: https://issues.apache.org/jira/browse/SPARK-14393 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.6.0 >Reporter: Jason Piper > > When utilising monotonicallyIncreasingId with a coalesce, it appears that > every partition uses the same offset (0) leading to non-monotonically > increasing IDs. > See examples below > ``` > >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show() > +---+ > |monotonicallyincreasingid()| > +---+ > |25769803776| > |51539607552| > |77309411328| > | 103079215104| > | 128849018880| > | 163208757248| > | 188978561024| > | 214748364800| > | 240518168576| > | 266287972352| > +---+ > ``` > >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show() > +---+ > |monotonicallyincreasingid()| > +---+ > | 0| > | 0| > |
[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce
[
https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Piper updated SPARK-14393:
Description:
When utilising monotonicallyIncreasingId with a coalesce, it appears that every
partition uses the same offset (0) leading to non-monotonically increasing IDs.
See examples below
{code}
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
+---+
|monotonicallyincreasingid()|
+---+
|25769803776|
|51539607552|
|77309411328|
| 103079215104|
| 128849018880|
| 163208757248|
| 188978561024|
| 214748364800|
| 240518168576|
| 266287972352|
+---+
```
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
+---+
>>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 1|
| 0|
| 0|
| 1|
| 2|
| 3|
| 0|
| 1|
| 2|
+---+
{code}
was:
When utilising monotonicallyIncreasingId with a coalesce, it appears that every
partition uses the same offset (0) leading to non-monotonically increasing IDs.
See examples below
```
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
+---+
|monotonicallyincreasingid()|
+---+
|25769803776|
|51539607552|
|77309411328|
| 103079215104|
| 128849018880|
| 163208757248|
| 188978561024|
| 214748364800|
| 240518168576|
| 266287972352|
+---+
```
>>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
| 0|
+---+
>>> sqlContext.range(10).repartition(5).select(monotonicallyIncreasingId()).coalesce(1).show()
+---+
|monotonicallyincreasingid()|
+---+
| 0|
| 1|
| 0|
| 0|
| 1|
| 2|
| 3|
| 0|
| 1|
| 2|
+---+
> monotonicallyIncreasingId not monotonically increasing with downstream
> coalesce
> ---
>
> Key: SPARK-14393
> URL: https://issues.apache.org/jira/browse/SPARK-14393
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Jason Piper
>
> When utilising monotonicallyIncreasingId with a coalesce, it appears that
> every partition uses the same offset (0) leading to non-monotonically
> increasing IDs.
> See examples below
> {code}
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> |25769803776|
> |51539607552|
> |77309411328|
> | 103079215104|
> | 128849018880|
> | 163208757248|
> | 188978561024|
> | 214748364800|
> | 240518168576|
> | 266287972352|
> +---+
> ```
> >>> sqlContext.range(10).select(monotonicallyIncreasingId()).coalesce(1).show()
> +---+
> |monotonicallyincreasingid()|
> +---+
> | 0|
> |
