pnikic-db commented on code in PR #55828:
URL: https://github.com/apache/spark/pull/55828#discussion_r3241470949
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -1954,6 +1954,35 @@ def test_window_functions(self):
for r, ex in zip(rs, expected):
self.assertEqual(tuple(r), ex[: len(r)])
+ def test_counter_diff_window_function(self):
+ df = self.spark.createDataFrame(
+ [
+ (1, datetime.datetime(2024, 1, 1), 100),
+ (2, datetime.datetime(2024, 1, 1), 200),
+ (3, datetime.datetime(2024, 1, 1), 400),
+ (4, datetime.datetime(2024, 1, 2), 50),
+ (5, datetime.datetime(2024, 1, 2), 150),
+ ],
+ ["t", "st", "c"],
+ )
+ w = Window.orderBy("t")
+
+ rows = df.select("t",
F.counter_diff("c").over(w).alias("d")).orderBy("t").collect()
+ self.assertEqual(
+ [(r.t, r.d) for r in rows],
+ [(1, None), (2, 100), (3, 200), (4, None), (5, 100)],
+ )
+
+ rows = (
+ df.select("t", F.counter_diff("c",
startTime="st").over(w).alias("d"))
+ .orderBy("t")
+ .collect()
+ )
+ self.assertEqual(
+ [(r.t, r.d) for r in rows],
+ [(1, None), (2, 100), (3, 200), (4, None), (5, 100)],
+ )
Review Comment:
I've changed the test to include both reset types separately: one for
counter value decrease and one for start time increase. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
