Yang Jie created SPARK-46681:
--------------------------------
Summary: Refactor `ExecutorFailureTracker#maxNumExecutorFailures`
to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured
Key: SPARK-46681
URL: https://issues.apache.org/jira/browse/SPARK-46681
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Yang Jie
{code:java}
def maxNumExecutorFailures(sparkConf: SparkConf): Int = {
val effectiveNumExecutors =
if (Utils.isStreamingDynamicAllocationEnabled(sparkConf)) {
sparkConf.get(STREAMING_DYN_ALLOCATION_MAX_EXECUTORS)
} else if (Utils.isDynamicAllocationEnabled(sparkConf)) {
sparkConf.get(DYN_ALLOCATION_MAX_EXECUTORS)
} else {
sparkConf.get(EXECUTOR_INSTANCES).getOrElse(0)
}
// By default, effectiveNumExecutors is Int.MaxValue if dynamic allocation is
enabled. We need
// avoid the integer overflow here.
val defaultMaxNumExecutorFailures = math.max(3,
if (effectiveNumExecutors > Int.MaxValue / 2) Int.MaxValue else 2 *
effectiveNumExecutors)
sparkConf.get(MAX_EXECUTOR_FAILURES).getOrElse(defaultMaxNumExecutorFailures)
} {code}
The result of defaultMaxNumExecutorFailures is calculated first, even if
{{MAX_EXECUTOR_FAILURES}} is configured now
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
