LeeeeLiu created SPARK-40377:
--------------------------------
Summary: Allow customize maxBroadcastTableBytes and
maxBroadcastRows
Key: SPARK-40377
URL: https://issues.apache.org/jira/browse/SPARK-40377
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.4.0
Reporter: LeeeeLiu
Recently, we encountered some driver OOM problems. Some tables with large data
volume were compressed using Snappy and then broadcast join was performed, but
the actual data volume was too large, which resulted in driver OOM.
The values of maxBroadcastTableBytes and maxBroadcastRows are hardcoded, 8GB
and 512000000 respectively. Maybe we can allow customization of these values,
configure smaller values according to different scenarios, and prohibit
broadcast joins for tables with large data volumes to avoid driver OOM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
