Kent Yao created SPARK-56948:
--------------------------------
Summary: TPCDSQueryBenchmark: make hardcoded SparkConf overridable
via setIfMissing
Key: SPARK-56948
URL: https://issues.apache.org/jira/browse/SPARK-56948
Project: Spark
Issue Type: Improvement
Components: Tests
Affects Versions: 5.0.0
Reporter: Kent Yao
Currently TPCDSQueryBenchmark hardcodes spark.driver.memory=3g,
spark.executor.memory=3g, and spark.sql.autoBroadcastJoinThreshold=20MB via
.set(...), which takes precedence over -Dspark.* system properties.
This prevents tuning heap/broadcast when running the benchmark at larger scale
factors (SF10 / SF100 / SF1000), where the 3g default OOMs.
spark.sql.shuffle.partitions already supports override via System.getProperty
in the same file -- extend the same pattern (via setIfMissing for consistency)
to the remaining three keys. Defaults unchanged, fully backward compatible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
