Yuming Wang created SPARK-32268: ----------------------------------- Summary: Bloom Filter Join Key: SPARK-32268 URL: https://issues.apache.org/jira/browse/SPARK-32268 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 3.1.0 Reporter: Yuming Wang Assignee: Yuming Wang Attachments: q16-bloom-filter.png, q16-default.png
We can improve the performance of some joins by pre-filtering one side of a join using a Bloom filter and IN predicate generated from the values from the other side of the join. For example:[tpcds/q16.sql|https://github.com/apache/spark/blob/a78d6ce376edf2a8836e01f47b9dff5371058d4c/sql/core/src/test/resources/tpcds/q16.sql] Before this optimization: After this optimization: *Query Performance Benchmarks: TPC-DS Performance Evaluation* Our setup for running TPC-DS benchmark was as follows: TPC-DS 5T and Partitioned Parquet table |Query|Default(Seconds)|Enable Bloom Filter Join(Seconds)| |tpcds q16|84|46| |tpcds q36|29|21| |tpcds q57|39|28| |tpcds q94|42|34| |tpcds q95|306|288| -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org