[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17451283#comment-17451283 ]
Nicholas Chammas edited comment on SPARK-26589 at 11/30/21, 6:17 PM: --------------------------------------------------------------------- I think there is a potential solution using the algorithm [described here by Michael Harris|https://www.quora.com/Distributed-Algorithms/What-is-the-distributed-algorithm-to-determine-the-median-of-arrays-of-integers-located-on-different-computers]. was (Author: nchammas): I'm going to try to implement this using the algorithm [described here by Michael Harris|https://www.quora.com/Distributed-Algorithms/What-is-the-distributed-algorithm-to-determine-the-median-of-arrays-of-integers-located-on-different-computers]. > proper `median` method for spark dataframe > ------------------------------------------ > > Key: SPARK-26589 > URL: https://issues.apache.org/jira/browse/SPARK-26589 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 3.1.0 > Reporter: Jan Gorecki > Priority: Minor > > I found multiple tickets asking for median function to be implemented in > Spark. Most of those tickets links to "SPARK-6761 Approximate quantile" as > duplicate of it. The thing is that approximate quantile is a workaround for > lack of median function. Thus I am filling this Feature Request for proper, > exact, not approximation of, median function. I am aware about difficulties > that are caused by distributed environment when trying to compute median, > nevertheless I don't think those difficulties is reason good enough to drop > out `median` function from scope of Spark. I am not asking about efficient > median but exact median. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org