LeeeeLiu created SPARK-40377: -------------------------------- Summary: Allow customize maxBroadcastTableBytes and maxBroadcastRows Key: SPARK-40377 URL: https://issues.apache.org/jira/browse/SPARK-40377 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.4.0 Reporter: LeeeeLiu
Recently, we encountered some driver OOM problems. Some tables with large data volume were compressed using Snappy and then broadcast join was performed, but the actual data volume was too large, which resulted in driver OOM. The values of maxBroadcastTableBytes and maxBroadcastRows are hardcoded, 8GB and 512000000 respectively. Maybe we can allow customization of these values, configure smaller values according to different scenarios, and prohibit broadcast joins for tables with large data volumes to avoid driver OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org