Yuming Wang created SPARK-32628:
-----------------------------------
Summary: Use bloom filter to improve dynamicPartitionPruning
Key: SPARK-32628
URL: https://issues.apache.org/jira/browse/SPARK-32628
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Yuming Wang
It will throw exception whenÂ
{{spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly}} is disabled:
{code:sql}
select catalog_sales.* from catalog_sales join catalog_returns where
cr_order_number = cs_sold_date_sk and cr_returned_time_sk < 40000;
{code}
{noformat}
20/08/16 06:44:42 ERROR TaskSetManager: Total size of serialized results of 494
tasks (1225.3 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)
{noformat}
We can improve it with minimum, maximum and bloom filter to reduce serialized
results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
