[
https://issues.apache.org/jira/browse/SPARK-33557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239631#comment-17239631
]
Yang Jie commented on SPARK-33557:
----------------------------------
It seems that changing value of "spark.network.timeout" doesn't really change
the value of STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT and their relationship is
maintained by code.
For example, the treatment of "spark.shuffle.io.connectionTimeout" is as
follows:
{code:java}
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
long defaultNetworkTimeoutS = JavaUtils.timeStringAsSec(
conf.get("spark.network.timeout", "120s"));
long defaultTimeoutMs = JavaUtils.timeStringAsSec(
conf.get(SPARK_NETWORK_IO_CONNECTIONTIMEOUT_KEY, defaultNetworkTimeoutS +
"s")) * 1000;
return (int) defaultTimeoutMs;
}
{code}
But it seems that there is no similar treatment
forSTORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT in HeartbeatReceiver and
MesosCoarseGrainedSchedulerBackend
{code:java}
private val executorTimeoutMs =
sc.conf.get(config.STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT)
{code}
{code:java}
mesosExternalShuffleClient.get
.registerDriverWithShuffleService(
agent.hostname,
externalShufflePort,
sc.conf.get(config.STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT),
sc.conf.get(config.EXECUTOR_HEARTBEAT_INTERVAL))
{code}
Maybe need to be fixed by code changes.
> spark.storage.blockManagerSlaveTimeoutMs default value does not follow
> spark.network.timeout value when the latter was changed
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-33557
> URL: https://issues.apache.org/jira/browse/SPARK-33557
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0, 3.0.1
> Reporter: Ohad
> Priority: Minor
>
> According to the documentation "spark.network.timeout" is the default timeout
> for "spark.storage.blockManagerSlaveTimeoutMs" which implies that when the
> user sets "spark.network.timeout" the effective value of
> "spark.storage.blockManagerSlaveTimeoutMs" should also be changed if it was
> not specifically changed.
> However this is not the case since the default value of
> "spark.storage.blockManagerSlaveTimeoutMs" is always the default value of
> "spark.network.timeout" (120s)
>
> "spark.storage.blockManagerSlaveTimeoutMs" is defined in the package object
> of "org.apache.spark.internal.config" as follows:
> {code:java}
> private[spark] val STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT =
> ConfigBuilder("spark.storage.blockManagerSlaveTimeoutMs")
> .version("0.7.0")
> .timeConf(TimeUnit.MILLISECONDS)
> .createWithDefaultString(Network.NETWORK_TIMEOUT.defaultValueString)
> {code}
> So it seems like the its default value is indeed "fixed" to
> "spark.network.timeout" default value.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
