[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13127034#comment-13127034 ] Suresh Srinivas commented on HDFS-1836: --- This patch is not required for trunk as it has equivalent code. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Assignee: Bharath Mundlapudi Fix For: 0.20.3, 0.20.205.0 Attachments: hdfs-1836-0.20.205.txt, hdfs-1836-0.20.txt, hdfs-1836-0.20.txt, patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071171#comment-13071171 ] Nathan Roberts commented on HDFS-1836: -- @Matt, I think this should be marked as fixed in 0.20.205.0. Is that correct? Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Assignee: Bharath Mundlapudi Fix For: 0.20.3, 0.20.204.0 Attachments: hdfs-1836-0.20.205.txt, hdfs-1836-0.20.txt, hdfs-1836-0.20.txt, patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13034554#comment-13034554 ] Hadoop QA commented on HDFS-1836: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479218/hdfs-1836-0.20.txt against trunk revision 1103987. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/536//console This message is automatically generated. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Attachments: hdfs-1836-0.20.txt, hdfs-1836-0.20.txt, patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13033466#comment-13033466 ] Bharath Mundlapudi commented on HDFS-1836: -- Thats correct. This code is part of trunk already. Todd, One minor comment. 1. Can we also pass LOG object to this method? Users who wants to debug can enable debug option. IOUtils.cleanup(LOG, blockStream, blockReplyStream); Otherwise, patch looks good. Thank you. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Attachments: hdfs-1836-0.20.txt, patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029794#comment-13029794 ] Hadoop QA commented on HDFS-1836: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478378/patch-draft-1836.patch against trunk revision 1100054. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/463//console This message is automatically generated. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Attachments: patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[
https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029802#comment-13029802
]
Bharath Mundlapudi commented on HDFS-1836:
--
Dennis,
For me, the following code seems like an issue.
try {
blockStream.close();
blockReplyStream.close();
} catch (IOException e) {
}
Reason: if blockStream throws an exception, blockReplyStream will not be closed.
Can we replace all the places (2 places) in DFSClient with the following and
try?
try {
blockStream.close();
} catch (IOException e) {
}
try {
blockReplyStream.close();
} catch (IOException e) {
}
Can you just try this change in your environment?
Thousand of CLOSE_WAIT socket
--
Key: HDFS-1836
URL: https://issues.apache.org/jira/browse/HDFS-1836
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.20.2
Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST
2011 x86_64 x86_64 x86_64 GNU/Linux
java version 1.6.0_23
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
Reporter: Dennis Cheung
Attachments: patch-draft-1836.patch
$ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT
4471
It is better if everything runs normal.
However, from time to time there are some DataStreamer Exception:
java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) |
Error Recovery for can be found from log file and the number of CLOSE_WAIT
socket just keep increasing
The CLOSE_WAIT handles may remain for hours and days; then Too many open
file some day.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[
https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029805#comment-13029805
]
Bharath Mundlapudi commented on HDFS-1836:
--
Retyping with format for better readability.
Change the following code
{code}
try {
blockStream.close();
blockReplyStream.close();
} catch (IOException e) {
}
{code}
to this:
{code}
try {
blockStream.close();
} catch (IOException e) {
}
try {
blockReplyStream.close();
} catch (IOException e) {
}
{code}
Thousand of CLOSE_WAIT socket
--
Key: HDFS-1836
URL: https://issues.apache.org/jira/browse/HDFS-1836
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.20.2
Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST
2011 x86_64 x86_64 x86_64 GNU/Linux
java version 1.6.0_23
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
Reporter: Dennis Cheung
Attachments: patch-draft-1836.patch
$ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT
4471
It is better if everything runs normal.
However, from time to time there are some DataStreamer Exception:
java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) |
Error Recovery for can be found from log file and the number of CLOSE_WAIT
socket just keep increasing
The CLOSE_WAIT handles may remain for hours and days; then Too many open
file some day.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029913#comment-13029913 ] Todd Lipcon commented on HDFS-1836: --- How about using IOUtils.cleanup, which does the above but also logs errors, etc? Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Attachments: patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13029950#comment-13029950 ] Bharath Mundlapudi commented on HDFS-1836: -- +1 to Todd's suggestion. Though current code logs errors in debug mode only. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung Attachments: patch-draft-1836.patch $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025084#comment-13025084 ] Dennis Cheung commented on HDFS-1836: - It creates a singleton FileSystem object which is connected to a HDFS cloud. And it does nothing with hadoop but one simple thing, copy local files into hadoop. We use only simple FileSystem.create(path, true) and IOUtils.copyLarge(). And already applied IOUtils.closeQuietly(out); in the finally block. Nothing more than that. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025592#comment-13025592 ] Dennis Cheung commented on HDFS-1836: - Two new finding after reading the hadoop source. 1. If there were an exception between lines DFSClient.java:2883-2901. The reference to blockReplyStream may lost and remain unclosed condition. 2. processDatanodeError() will not close the previous Socket, if any. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025030#comment-13025030 ] Dennis Cheung commented on HDFS-1836: - @Allen Wittenauer Update. None. `netstat -ano` shown NO any connection to the client. But only a few ESTABLISHED connection from some other datanode / clients Recheck on the client side, the connection may still remain CLOSE_WAIT. Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025061#comment-13025061 ] Bharath Mundlapudi commented on HDFS-1836: -- Hi Dennis, What is this client code doing? Is this your program which uses HDFS APIs to talk to datanode and namenode? -Bharath Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020736#comment-13020736 ] Dennis Cheung commented on HDFS-1836: - @Allen Wittenauer Unknown, I've no access to the remote system Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1836) Thousand of CLOSE_WAIT socket
[ https://issues.apache.org/jira/browse/HDFS-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13019927#comment-13019927 ] Allen Wittenauer commented on HDFS-1836: What is the state of the other side of the socket connection? Thousand of CLOSE_WAIT socket -- Key: HDFS-1836 URL: https://issues.apache.org/jira/browse/HDFS-1836 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20.2 Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_23 Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Reporter: Dennis Cheung $ /usr/sbin/lsof -i TCP:50010 | grep -c CLOSE_WAIT 4471 It is better if everything runs normal. However, from time to time there are some DataStreamer Exception: java.net.SocketTimeoutException and DFSClient.processDatanodeError(2507) | Error Recovery for can be found from log file and the number of CLOSE_WAIT socket just keep increasing The CLOSE_WAIT handles may remain for hours and days; then Too many open file some day. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
