[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536711#comment-14536711
]
Hudson commented on HDFS-8311:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536664#comment-14536664
]
Hudson commented on HDFS-8311:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #190 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/190/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536604#comment-14536604
]
Hudson commented on HDFS-8311:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #180 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/180/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536553#comment-14536553
]
Hudson commented on HDFS-8311:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2120 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2120/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536460#comment-14536460
]
Hudson commented on HDFS-8311:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #922 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/922/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536409#comment-14536409
]
Hudson commented on HDFS-8311:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #191 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/191/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535589#comment-14535589
]
Esteban Gutierrez commented on HDFS-8311:
-
[~brahmareddy], [~yzhangal] I've created HDFS-8359. Depending what we might
find down the road we can move that jira as an umbrella jura and this jira as a
subtask of HDFS-8359. Thanks again!
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535570#comment-14535570
]
Yongjun Zhang commented on HDFS-8311:
-
HI [~brahmareddy],
Thanks for your interest, I was about to check with [~esteban], so I will let
you guys decide who will fix it. Thanks.
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535519#comment-14535519
]
Brahma Reddy Battula commented on HDFS-8311:
[~yzhangal]
Separate is logged for following..? If not, can I file jira and fix..?
{{DataXceiver#replaceBlock}}
{code} OutputStream unbufProxyOut = NetUtils.getOutputStream(proxySock,
dnConf.socketWriteTimeout);
InputStream unbufProxyIn = NetUtils.getInputStream(proxySock);{code}
{{DataXceiver#writeBlock}}
{code} OutputStream unbufMirrorOut =
NetUtils.getOutputStream(mirrorSock,
writeTimeout);
InputStream unbufMirrorIn = NetUtils.getInputStream(mirrorSock);{code}
{{DataNode#run}}
{code}OutputStream unbufOut = NetUtils.getOutputStream(sock,
writeTimeout);
InputStream unbufIn = NetUtils.getInputStream(sock);{code}
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535514#comment-14535514
]
Esteban Gutierrez commented on HDFS-8311:
-
Thanks [~yzhangal]!
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Fix For: 2.8.0
>
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535501#comment-14535501
]
Yongjun Zhang commented on HDFS-8311:
-
Thanks [~esteban] for reporting the issue and the fix, thanks Colin for the
review, I committed to trunk and branch-2.
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Labels: BB2015-05-TBR
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535440#comment-14535440
]
Hudson commented on HDFS-8311:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7778 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7778/])
HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream.
(Esteban Gutierrez via Yongjun Zhang) (yzhang: rev
730f9930a48259f34e48404aee51e8d641cc3d36)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Labels: BB2015-05-TBR
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535277#comment-14535277
]
Yongjun Zhang commented on HDFS-8311:
-
Thanks [~cmccabe]. +1 and will commit soon.
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Labels: BB2015-05-TBR
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535272#comment-14535272
]
Colin Patrick McCabe commented on HDFS-8311:
Good catch, [~yzhangal]. We should fix those other cases as well. I think we
should do those in separate JIRAs, if that's more convenient for you. Also, it
would be nice to have unit tests for these timeouts at some point, to ensure
that they don't get removed.
+1 again for the patch. Thanks, guys.
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Labels: BB2015-05-TBR
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535104#comment-14535104
]
Yongjun Zhang commented on HDFS-8311:
-
Hi [~esteban],
Thanks for addressing my verbal comments.
I found that there are other places that might need similar handling, like
{{DataXceiver#replaceBlock}}
{code}
OutputStream unbufProxyOut = NetUtils.getOutputStream(proxySock,
dnConf.socketWriteTimeout);
InputStream unbufProxyIn = NetUtils.getInputStream(proxySock);
{code}
{{DataXceiver#writeBlock}}
{code}
OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock,
writeTimeout);
InputStream unbufMirrorIn = NetUtils.getInputStream(mirrorSock);
{code}
{{DataNode#run}}
{code}
OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
InputStream unbufIn = NetUtils.getInputStream(sock);
{code}
And some other places.
I know that [~esteban] has verified the uploaded patch solve one real issue, so
maybe it's ok for us to address the remaining places in separate jiras.
What do you think [~cmccabe]?
Thanks.
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Labels: BB2015-05-TBR
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529413#comment-14529413
]
Hadoop QA commented on HDFS-8311:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 15s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 0s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 167m 45s | Tests passed in hadoop-hdfs.
|
| | | 210m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730229/HDFS-8311.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ffce9a3 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10818/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10818/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10818/console |
This message was automatically generated.
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529027#comment-14529027
]
Colin Patrick McCabe commented on HDFS-8311:
Thanks, [~esteban]. +1 pending jenkins
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8311) DataStreamer.transfer() should timeout the socket InputStream.
[
https://issues.apache.org/jira/browse/HDFS-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528057#comment-14528057
]
Kiran Kumar M R commented on HDFS-8311:
---
Thats fine, reassigned
> DataStreamer.transfer() should timeout the socket InputStream.
> --
>
> Key: HDFS-8311
> URL: https://issues.apache.org/jira/browse/HDFS-8311
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
> Attachments:
> 0001-HDFS-8311-DataStreamer.transfer-should-timeout-the-s.patch,
> HDFS-8311.001.patch
>
>
> While validating some HA failure modes we found that HDFS clients can take a
> long time to recover or sometimes don't recover at all since we don't setup
> the socket timeout in the InputStream:
> {code}
> private void transfer () { ...
> ...
> OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
> InputStream unbufIn = NetUtils.getInputStream(sock);
> ...
> }
> {code}
> The InputStream should have its own timeout in the same way as the
> OutputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
