[
https://issues.apache.org/jira/browse/SPARK-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501063#comment-14501063
]
Aaron Davidson commented on SPARK-6962:
---------------------------------------
I have a hypothesis that the above is caused by assumptions we make about the
eventuality/symmetry of socket timeouts that are not guaranteed in arbitrary
network topologies. If this is the case, though, then I would also expect nio
to have intermittent failures, though it could at least recover from them.
A potential fix would be a timer thread in
[TransportResponseHandler|https://github.com/apache/spark/blob/master/network/common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java]
which would require that there is some message received every N seconds (the
default network timeout in Spark is 120 seconds) as long as there is some
outstanding request. This should be fairly robust due to our use of retries on
IOExceptions in
[RetryingBlockFetcher|https://github.com/apache/spark/blob/master/network/shuffle/src/main/java/org/apache/spark/network/shuffle/RetryingBlockFetcher.java].
> Netty BlockTransferService hangs in the middle of SQL query
> -----------------------------------------------------------
>
> Key: SPARK-6962
> URL: https://issues.apache.org/jira/browse/SPARK-6962
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.2.0, 1.2.1, 1.3.0
> Reporter: Jon Chase
> Attachments: jstacks.txt
>
>
> Spark SQL queries (though this seems to be a Spark Core issue - I'm just
> using queries in the REPL to surface this, so I mention Spark SQL) hang
> indefinitely under certain (not totally understood) circumstances.
> This is resolved by setting spark.shuffle.blockTransferService=nio, which
> seems to point to netty as the issue. Netty was set as the default for the
> block transport layer in 1.2.0, which is when this issue started. Setting
> the service to nio allows queries to complete normally.
> I do not see this problem when running queries over smaller (~20 5MB files)
> datasets. When I increase the scope to include more data (several hundred
> ~5MB files), the queries will get through several steps but eventuall hang
> indefinitely.
> Here's the email chain regarding this issue, including stack traces:
> http://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/<cae61spfqt2y7d5vqzomzz2dmr-jx2c2zggcyky40npkjjx4...@mail.gmail.com>
> For context, here's the announcement regarding the block transfer service
> change:
> http://mail-archives.apache.org/mod_mbox/spark-dev/201411.mbox/<cabpqxssl04q+rbltp-d8w+z3atn+g-um6gmdgdnh-hzcvd-...@mail.gmail.com>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
