gabotechs commented on PR #25337: URL: https://github.com/apache/datafusion/pull/25337#issuecomment-6059764003
> Also: this is [supported by spark](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.approx_percentile.html) Regarding this, I'm seeing that this is actually not supported by Spark. Did a quick test with a local Spark container running 4.0.1 and this is what I'm seeing: ``` | Percentile argument | PR behavior | Spark 4.0.1 | |---|---|---| | Literal or `0.25 + 0.25` | Accepted | Accepted | | Projected literal, `SELECT …, 0.5 AS m` | Accepted | `NON_FOLDABLE_INPUT` | | Column containing only `0.5` | Accepted | `NON_FOLDABLE_INPUT` | | Column constant within each group | Accepted | `NON_FOLDABLE_INPUT` | | Column with empty input | Returns `NULL` | `NON_FOLDABLE_INPUT` | ``` In the doc you linked https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.approx_percentile.html#pyspark-sql-functions-approx-percentile, it looks like that `Column` is not an actual "column" as known in relational algebra, is just a Python class that can reference to a literal or an expression. For reference [here is the Spark code](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala#L125-L133) that rejects these kind of queries in case of a NON_FOLDABLE_INPUT. Given that this is supported by Trino, and that it's still API compatible with Spark and Postgres, I'd still be in favor of merging this, however, I'm not comfortable just pulling this in without an extra pair of eyes from a PMC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
