RyanBerti opened a new pull request, #40615: URL: https://github.com/apache/spark/pull/40615
### What changes were proposed in this pull request? This PR adds a new dependency on the datasketches-java project, and provides 3 new functions which utilize Datasketches HllSketch and Union instances for unique counting and intermediate storage of HLL sketches. ### Why are the changes needed? The existing approx_count_distinct provides unique counting functionality, but does not provide the ability to store HLL sketches for re-aggregation. After discussions with the Databricks team, we decided that integrating the Datasketches HllSketch implementation would make more sense than extending the existing approx_count_distinct functionality, given the wide use of the Datasketches library. See https://github.com/apache/spark/pull/39678 and https://github.com/RyanBerti/spark/pull/1 for more information. ### Does this PR introduce _any_ user-facing change? Yes, this PR introduces three new aggregate functions: - hllsketch_estimate - hllsketch_binary - hllsketch_union_estimate ### How was this patch tested? <WIP> I've included a simple test in the DataframeAggregateSuite; more tests to be added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
