[ https://issues.apache.org/jira/browse/SPARK-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063211#comment-14063211 ]
Aaron Davidson commented on SPARK-2282: --------------------------------------- This should actually only be necessary on the master. Use of the SO_REUSEADDR property (equivalently, sysctl tcp_tw_reuse) means that the number of used sockets will increase to the maximum number of ephemeral ports, but then should remain constant. It's possible that if another process tries to allocate an ephemeral port during this time, it will fail. While tcp_tw_reuse is generally considered "safe", setting *tcp_tw_recycle* can lead to unexpected packet arrival from closed streams (though it's very unlikely), but is a more guaranteed solution. This should cause the connections to be recycled immediately after the TCP teardown, and thus no buildup of sockets should occur. Please let me know if setting either of these parameters helps on the driver machine. You can also verify that this problem is occurring by doing a {{netstat -lpn}} during execution, iirc, which should display an inordinate number of open connections on the Spark Driver process and on a Python daemon one. > PySpark crashes if too many tasks complete quickly > -------------------------------------------------- > > Key: SPARK-2282 > URL: https://issues.apache.org/jira/browse/SPARK-2282 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 0.9.1, 1.0.0, 1.0.1 > Reporter: Aaron Davidson > Assignee: Aaron Davidson > Fix For: 0.9.2, 1.0.0, 1.0.1 > > > Upon every task completion, PythonAccumulatorParam constructs a new socket to > the Accumulator server running inside the pyspark daemon. This can cause a > buildup of used ephemeral ports from sockets in the TIME_WAIT termination > stage, which will cause the SparkContext to crash if too many tasks complete > too quickly. We ran into this bug with 17k tasks completing in 15 seconds. > This bug can be fixed outside of Spark by ensuring these properties are set > (on a linux server); > echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse > echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle > or by adding the SO_REUSEADDR option to the Socket creation within Spark. -- This message was sent by Atlassian JIRA (v6.2#6252)