[
https://issues.apache.org/jira/browse/SPARK-28239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deegue updated SPARK-28239:
---------------------------
Description:
When executing shuffle tasks, TCP connections(on port 7337 by default) will be
established by shuffle service.
It will like:
!screenshot-1.png!
However, some of the TCP connections are still busy when the task is actually
finished. These connections won't close automatically until we restart the
NodeManager process.
Connections pile up and NodeManagers are getting slower and slower.
!screenshot-2.png!
was:When executing shuffle tasks,
> Make TCP connections created by shuffle service auto close on YARN
> NodeManagers
> -------------------------------------------------------------------------------
>
> Key: SPARK-28239
> URL: https://issues.apache.org/jira/browse/SPARK-28239
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, YARN
> Affects Versions: 2.4.0
> Environment: Hadoop2.6.0-CDH5.8.3(netty3)
> Spark2.4.0(netty4)
> set spark.shuffle.service.enabled=true
> Reporter: Deegue
> Priority: Minor
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> When executing shuffle tasks, TCP connections(on port 7337 by default) will
> be established by shuffle service.
> It will like:
> !screenshot-1.png!
> However, some of the TCP connections are still busy when the task is actually
> finished. These connections won't close automatically until we restart the
> NodeManager process.
> Connections pile up and NodeManagers are getting slower and slower.
> !screenshot-2.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
