[ https://issues.apache.org/jira/browse/SPARK-41904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655629#comment-17655629 ]
jiaan.geng commented on SPARK-41904: ------------------------------------ [~techaddict]Could you tell me how to reproduce this issue? I want take a look! > Fix Function `nth_value` functions output > ----------------------------------------- > > Key: SPARK-41904 > URL: https://issues.apache.org/jira/browse/SPARK-41904 > Project: Spark > Issue Type: Sub-task > Components: Connect > Affects Versions: 3.4.0 > Reporter: Sandeep Singh > Priority: Major > > {code:java} > from pyspark.sql import Window > from pyspark.sql.functions import nth_value > df = self.spark.createDataFrame( > [ > ("a", 0, None), > ("a", 1, "x"), > ("a", 2, "y"), > ("a", 3, "z"), > ("a", 4, None), > ("b", 1, None), > ("b", 2, None), > ], > schema=("key", "order", "value"), > ) > w = Window.partitionBy("key").orderBy("order") > rs = df.select( > df.key, > df.order, > nth_value("value", 2).over(w), > nth_value("value", 2, False).over(w), > nth_value("value", 2, True).over(w), > ).collect() > expected = [ > ("a", 0, None, None, None), > ("a", 1, "x", "x", None), > ("a", 2, "x", "x", "y"), > ("a", 3, "x", "x", "y"), > ("a", 4, "x", "x", "y"), > ("b", 1, None, None, None), > ("b", 2, None, None, None), > ] > for r, ex in zip(sorted(rs), sorted(expected)): > self.assertEqual(tuple(r), ex[: len(r)]){code} > {code:java} > Traceback (most recent call last): > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", > line 755, in test_nth_value > self.assertEqual(tuple(r), ex[: len(r)]) > AssertionError: Tuples differ: ('a', 1, 'x', None) != ('a', 1, 'x', 'x') > First differing element 3: > None > 'x' > - ('a', 1, 'x', None) > ? ^^^^ > + ('a', 1, 'x', 'x') > ? ^^^ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org