[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17636782#comment-17636782
]
Nikhil Sharma commented on SPARK-35563:
---
Thank you for sharing such good information. Very informative and effective
post.
+[https://www.igmguru.com/digital-marketing-programming/ruby-on-rails-certification-training/]+
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627252#comment-17627252
]
Vivek Garg commented on SPARK-35563:
Alternately, simply perform the int overflow check. Personally, I don't think
it's a problem that Spark doesn't support really huge windows. I simply want to
prevent data loss or corruption.
Thank you.
[Salesforce Marketing Cloud
Certification|https://www.igmguru.com/salesforce/salesforce-marketing-cloud-training/]
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368098#comment-17368098
]
Robert Joseph Evans commented on SPARK-35563:
-
Or just do the overflow check on the int. I personally don't see a problem with
Spark not supporting really large windows. I just want to avoid data
corruption/loss.
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368096#comment-17368096
]
Robert Joseph Evans commented on SPARK-35563:
-
Yes, technically if we switch it from an int to a long then we will have a
similar problem with LONG_MAX. But that kicks the can down the road a very
long ways. With the current spark memory layout for unsafe row where there is a
long for nullability followed by a long for each column (possibly more) for a
single column dataframe we would need 32 exabytes of memory to hold this window
before we hit the problem. But yes we should look at doing an overflow check
as well. I just would want to measure the performance impact of it so we can
make an informed decision.
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367437#comment-17367437
]
vinay commented on SPARK-35563:
---
If we change the row_number from int to long is it not like pushing out the
problem from INT_MAX to LONG_MAX?
While it's a huge value to ever encounter in window size scenario, i think we
can alert the user in some way along with changing from int -> long.
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362910#comment-17362910
]
Robert Joseph Evans commented on SPARK-35563:
-
[~dc-heros] Thanks for looking into this. I was trying to understand how
`row_number` being an int would cause data loss, I don't care about the
overflow of the result for `row_number`. That does not feel like a bug to me,
as you said. But it pointed me in the right direction.
At first I thought that there must be an issue with the buffer, but as I looked
more closely I found a variable that does essentially the same thing as
`row_number`.
https://github.com/apache/spark/blob/439e94c1712366ff267183d3946f2507ebf3a98e/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExec.scala#L179
It is an int and is overflowing causing the iterator to terminate the loop
early. I think if we switch `rowNumber` to be a long the data loss problem will
go away. But it is not a super trivial change, because `WindowFunctionFrame`
takes that variable as input to `write` so we need to decide if we would rather
blow up when rowNumber overflows, or if we should change the index to be a long
and pull on that string until everything is updated.
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362824#comment-17362824
]
dgd_contributor commented on SPARK-35563:
-
After looking to this, I found out rowNumber in RowNumberLike is interger_type,
but I don't think is a bug? Should I create a pull request?
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35563) [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
[
https://issues.apache.org/jira/browse/SPARK-35563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362250#comment-17362250
]
Cao Jianghe commented on SPARK-35563:
-
I'd like to look into this issue.
> [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
> --
>
> Key: SPARK-35563
> URL: https://issues.apache.org/jira/browse/SPARK-35563
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.2
>Reporter: Robert Joseph Evans
>Priority: Blocker
> Labels: data-loss
>
> I think this impacts a lot more versions of Spark, but I don't know for sure
> because it takes a long time to test. As a part of doing corner case
> validation testing for spark rapids I found that if a window function has
> more than {{Int.MaxValue + 1}} rows the result is silently truncated to that
> many rows. I have only tested this on 3.0.2 with {{row_number}}, but I
> suspect it will impact others as well. This is a really rare corner case, but
> because it is silent data corruption I personally think it is quite serious.
> {code:scala}
> import org.apache.spark.sql.expressions.Window
> val windowSpec = Window.partitionBy("a").orderBy("b")
> val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as
> b")
> spark.time(df.select(col("a"), col("b"),
> row_number().over(windowSpec).alias("rn")).orderBy(desc("a"),
> desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))
> +-+--+
>
> | dir| count|
> +-+--+
> |false|2147483647|
> | true| 1|
> +-+--+
> Time taken: 1139089 ms
> Int.MaxValue.toLong + 100
> res15: Long = 2147483747
> 2147483647L + 1
> res16: Long = 2147483648
> {code}
> I had to make sure that I ran the above with at least 64GiB of heap for the
> executor (I did it in local mode and it worked, but took forever to run)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
