Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
zhengruifeng commented on PR #46456: URL: https://github.com/apache/spark/pull/46456#issuecomment-2101670472 thanks @dongjoon-hyun and @HyukjinKwon for reviews -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
dongjoon-hyun commented on PR #46456: URL: https://github.com/apache/spark/pull/46456#issuecomment-2100758425 Thank you, @zhengruifeng and @HyukjinKwon . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
dongjoon-hyun commented on PR #46456: URL: https://github.com/apache/spark/pull/46456#issuecomment-2100756927 Merged to master/3.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
dongjoon-hyun closed pull request #46456: [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side URL: https://github.com/apache/spark/pull/46456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
dongjoon-hyun commented on code in PR #46456:
URL: https://github.com/apache/spark/pull/46456#discussion_r1593374153
##
python/pyspark/sql/tests/test_dataframe.py:
##
@@ -430,6 +430,11 @@ def test_sample(self):
IllegalArgumentException, lambda:
self.spark.range(1).sample(-1.0).count()
)
+def test_sample_with_random_seed(self):
+df = self.spark.range(1).sample(0.1)
+cnts = [df.count() for i in range(10)]
+self.assertEqual(1, len(set(cnts)))
Review Comment:
Oh, it seems that we have a unit test.
```
FAIL [0.001s]: test_sample
(pyspark.sql.tests.connect.test_connect_plan.SparkConnectPlanTests.test_sample)
--
Traceback (most recent call last):
File
"/__w/spark/spark/python/pyspark/sql/tests/connect/test_connect_plan.py", line
446, in test_sample
self.assertEqual(plan.root.sample.HasField("seed"), False)
AssertionError: True != False
```
##
python/pyspark/sql/tests/test_dataframe.py:
##
@@ -430,6 +430,11 @@ def test_sample(self):
IllegalArgumentException, lambda:
self.spark.range(1).sample(-1.0).count()
)
+def test_sample_with_random_seed(self):
+df = self.spark.range(1).sample(0.1)
+cnts = [df.count() for i in range(10)]
+self.assertEqual(1, len(set(cnts)))
Review Comment:
Oh, it seems that we have a unit test. Could you update it?
```
FAIL [0.001s]: test_sample
(pyspark.sql.tests.connect.test_connect_plan.SparkConnectPlanTests.test_sample)
--
Traceback (most recent call last):
File
"/__w/spark/spark/python/pyspark/sql/tests/connect/test_connect_plan.py", line
446, in test_sample
self.assertEqual(plan.root.sample.HasField("seed"), False)
AssertionError: True != False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
dongjoon-hyun commented on code in PR #46456:
URL: https://github.com/apache/spark/pull/46456#discussion_r1593374153
##
python/pyspark/sql/tests/test_dataframe.py:
##
@@ -430,6 +430,11 @@ def test_sample(self):
IllegalArgumentException, lambda:
self.spark.range(1).sample(-1.0).count()
)
+def test_sample_with_random_seed(self):
+df = self.spark.range(1).sample(0.1)
+cnts = [df.count() for i in range(10)]
+self.assertEqual(1, len(set(cnts)))
Review Comment:
Oh, it seems that we have a unit test already. Shall we remove this and
update that one, @zhengruifeng , in order to avoid increasing test time?
```
FAIL [0.001s]: test_sample
(pyspark.sql.tests.connect.test_connect_plan.SparkConnectPlanTests.test_sample)
--
Traceback (most recent call last):
File
"/__w/spark/spark/python/pyspark/sql/tests/connect/test_connect_plan.py", line
446, in test_sample
self.assertEqual(plan.root.sample.HasField("seed"), False)
AssertionError: True != False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
dongjoon-hyun commented on PR #46456: URL: https://github.com/apache/spark/pull/46456#issuecomment-2099681039 Ah, I got your point. It's a very interesting `connector` bug. > df2 should be immutable. I was thinking the following. My bad. ``` scala> spark.range(1).sample(0.1).count() res0: Long = 1080 scala> spark.range(1).sample(0.1).count() res1: Long = 998 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
zhengruifeng commented on PR #46456: URL: https://github.com/apache/spark/pull/46456#issuecomment-2099676615 @dongjoon-hyun `df2 = df1.sample()`, the dataframe `df2` should be immutable. I think it is a bug, we should backport it, have update the affected versions in jira. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[PR] [SPARK-48184][PYTHON][CONNECT] Always set the seed of `Dataframe.sample` in Client side [spark]
zhengruifeng opened a new pull request, #46456: URL: https://github.com/apache/spark/pull/46456 ### What changes were proposed in this pull request? Always set the seed of `Dataframe.sample` in Client side ### Why are the changes needed? Bug fix If the seed is not set in Client, it will be set in server side with a random int https://github.com/apache/spark/blob/c4df12cc884cddefcfcf8324b4d7b9349fb4f6a0/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L386 which cause inconsistent results in multiple executions In Spark Classic: ``` In [1]: df = spark.range(1).sample(0.1) In [2]: [df.count() for i in range(10)] Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006] ``` In Spark Connect: before: ``` In [1]: df = spark.range(1).sample(0.1) In [2]: [df.count() for i in range(10)] Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979] ``` after: ``` In [1]: df = spark.range(1).sample(0.1) In [2]: [df.count() for i in range(10)] Out[2]: [1032, 1032, 1032, 1032, 1032, 1032, 1032, 1032, 1032, 1032] ``` ### Does this PR introduce _any_ user-facing change? yes, bug fix ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
