[ 
https://issues.apache.org/jira/browse/SPARK-33557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239631#comment-17239631
 ] 

Yang Jie commented on SPARK-33557:
----------------------------------

It seems that changing value of "spark.network.timeout" doesn't really change 
the value of STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT and their relationship is 
maintained by code.

For example, the treatment of "spark.shuffle.io.connectionTimeout" is as 
follows:

 
{code:java}
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
  long defaultNetworkTimeoutS = JavaUtils.timeStringAsSec(
    conf.get("spark.network.timeout", "120s"));
  long defaultTimeoutMs = JavaUtils.timeStringAsSec(
    conf.get(SPARK_NETWORK_IO_CONNECTIONTIMEOUT_KEY, defaultNetworkTimeoutS + 
"s")) * 1000;
  return (int) defaultTimeoutMs;
}
{code}
 

 

But it seems that there is no similar treatment 
forSTORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT in HeartbeatReceiver and 
MesosCoarseGrainedSchedulerBackend

 
{code:java}
private val executorTimeoutMs = 
sc.conf.get(config.STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT)
{code}
 

 
{code:java}
mesosExternalShuffleClient.get
  .registerDriverWithShuffleService(
    agent.hostname,
    externalShufflePort,
    sc.conf.get(config.STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT),
    sc.conf.get(config.EXECUTOR_HEARTBEAT_INTERVAL))
{code}
 

Maybe need to be fixed by code changes.

 

 

> spark.storage.blockManagerSlaveTimeoutMs default value does not follow 
> spark.network.timeout value when the latter was changed
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-33557
>                 URL: https://issues.apache.org/jira/browse/SPARK-33557
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.0.0, 3.0.1
>            Reporter: Ohad
>            Priority: Minor
>
> According to the documentation "spark.network.timeout" is the default timeout 
> for "spark.storage.blockManagerSlaveTimeoutMs" which implies that when the 
> user sets "spark.network.timeout"  the effective value of 
> "spark.storage.blockManagerSlaveTimeoutMs" should also be changed if it was 
> not specifically changed.
> However this is not the case since the default value of 
> "spark.storage.blockManagerSlaveTimeoutMs" is always the default value of 
> "spark.network.timeout" (120s)
>  
> "spark.storage.blockManagerSlaveTimeoutMs" is defined in the package object 
> of "org.apache.spark.internal.config" as follows:
> {code:java}
> private[spark] val STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT =
>   ConfigBuilder("spark.storage.blockManagerSlaveTimeoutMs")
>     .version("0.7.0")
>     .timeConf(TimeUnit.MILLISECONDS)
>     .createWithDefaultString(Network.NETWORK_TIMEOUT.defaultValueString)
> {code}
> So it seems like the its default value is indeed "fixed" to 
> "spark.network.timeout" default value.
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to