Xinrong Meng created SPARK-36310: ------------------------------------ Summary: Fix hasnan(), any(), and all() window function Key: SPARK-36310 URL: https://issues.apache.org/jira/browse/SPARK-36310 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 3.2.0 Reporter: Xinrong Meng
{code:java} File "/__w/spark/spark/python/pyspark/pandas/groupby.py", line 1497, in pyspark.pandas.groupby.GroupBy.rank Failed example: df.groupby("a").rank().sort_index() Exception raised: ... pyspark.sql.utils.AnalysisException: It is not allowed to use a window function inside an aggregate function. Please use the inner window function in a sub-query. {code} As shown above, hasnans() used in "rank" causes "It is not allowed to use a window function inside an aggregate function" exception. any() and all() have the same issue. We shall adjust that. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org