[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293598#comment-14293598 ] Hudson commented on HDFS-7224: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2018 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2018/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293642#comment-14293642 ] Hudson commented on HDFS-7224: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #87 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/87/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293566#comment-14293566 ] Hudson commented on HDFS-7224: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #83 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/83/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293575#comment-14293575 ] Hudson commented on HDFS-7224: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2037 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2037/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293313#comment-14293313 ] Hudson commented on HDFS-7224: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #86 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/86/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293325#comment-14293325 ] Hudson commented on HDFS-7224: -- FAILURE: Integrated in Hadoop-Yarn-trunk #820 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/820/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291862#comment-14291862 ] Kihwal Lee commented on HDFS-7224: -- I've committed this to branch-2 and trunk. Thanks for working on this, Eric, and for the review, Daryn. Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291875#comment-14291875 ] Hudson commented on HDFS-7224: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6930 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6930/]) HDFS-7224. Allow reuse of NN connections via webhdfs. Contributed by Eric Payne (kihwal: rev 2b0fa20f69417326a92beac10ffa072db2616e73) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 2.7.0 Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[ https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287540#comment-14287540 ] Daryn Sharp commented on HDFS-7224: --- +1 Looks good to me. We've been running in production with this patch since last year. Allow reuse of NN connections via webhdfs - Key: HDFS-7224 URL: https://issues.apache.org/jira/browse/HDFS-7224 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.5.0 Reporter: Eric Payne Assignee: Eric Payne Attachments: HDFS-7224.v1.201410301923.txt, HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt In very large clusters, the webhdfs client could get bind exceptions because it runs out of ephemeral ports. This could happen when using webhdfs to talk to the NN in order to do list globbing of a huge amount of files. WebHdfsFileSystem#jsonParse gets the input/error stream from the connection, but never closes the stream. Since it's not closed, the JVM thinks the stream may still be transferring data, so the next time through this code, it has to get a new connection rather than reusing an existing one. The lack of connection reuse has poor latency and adds too much overhead to the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280666#comment-14280666
]
Hadoop QA commented on HDFS-7224:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692811/HDFS-7224.v3.txt
against trunk revision ec4389c.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9251//console
This message is automatically generated.
Allow reuse of NN connections via webhdfs
-
Key: HDFS-7224
URL: https://issues.apache.org/jira/browse/HDFS-7224
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.0
Reporter: Eric Payne
Assignee: Eric Payne
Attachments: HDFS-7224.v1.201410301923.txt,
HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt
In very large clusters, the webhdfs client could get bind exceptions because
it runs out of ephemeral
ports. This could happen when using webhdfs to talk to the NN in order to do
list globbing of a
huge amount of files.
WebHdfsFileSystem#jsonParse gets the input/error stream from the connection,
but never closes the stream. Since it's not closed, the JVM thinks the stream
may still
be transferring data, so the next time through this code, it has to get a new
connection
rather than reusing an existing one.
The lack of connection reuse has poor latency and adds too much overhead to
the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281007#comment-14281007
]
Hadoop QA commented on HDFS-7224:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692828/HDFS-7224.v4.txt
against trunk revision ec4389c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9254//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9254//console
This message is automatically generated.
Allow reuse of NN connections via webhdfs
-
Key: HDFS-7224
URL: https://issues.apache.org/jira/browse/HDFS-7224
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.0
Reporter: Eric Payne
Assignee: Eric Payne
Attachments: HDFS-7224.v1.201410301923.txt,
HDFS-7224.v2.201410312033.txt, HDFS-7224.v3.txt, HDFS-7224.v4.txt
In very large clusters, the webhdfs client could get bind exceptions because
it runs out of ephemeral
ports. This could happen when using webhdfs to talk to the NN in order to do
list globbing of a
huge amount of files.
WebHdfsFileSystem#jsonParse gets the input/error stream from the connection,
but never closes the stream. Since it's not closed, the JVM thinks the stream
may still
be transferring data, so the next time through this code, it has to get a new
connection
rather than reusing an existing one.
The lack of connection reuse has poor latency and adds too much overhead to
the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192760#comment-14192760
]
Hadoop QA commented on HDFS-7224:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678567/HDFS-7224.v2.201410312033.txt
against trunk revision 1b4be91.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8619//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8619//console
This message is automatically generated.
Allow reuse of NN connections via webhdfs
-
Key: HDFS-7224
URL: https://issues.apache.org/jira/browse/HDFS-7224
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.0
Reporter: Eric Payne
Assignee: Eric Payne
Attachments: HDFS-7224.v1.201410301923.txt,
HDFS-7224.v2.201410312033.txt
In very large clusters, the webhdfs client could get bind exceptions because
it runs out of ephemeral
ports. This could happen when using webhdfs to talk to the NN in order to do
list globbing of a
huge amount of files.
WebHdfsFileSystem#jsonParse gets the input/error stream from the connection,
but never closes the stream. Since it's not closed, the JVM thinks the stream
may still
be transferring data, so the next time through this code, it has to get a new
connection
rather than reusing an existing one.
The lack of connection reuse has poor latency and adds too much overhead to
the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7224) Allow reuse of NN connections via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14190671#comment-14190671
]
Hadoop QA commented on HDFS-7224:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678284/HDFS-7224.v1.201410301923.txt
against trunk revision c2866ac.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8600//console
This message is automatically generated.
Allow reuse of NN connections via webhdfs
-
Key: HDFS-7224
URL: https://issues.apache.org/jira/browse/HDFS-7224
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.0
Reporter: Eric Payne
Assignee: Eric Payne
Attachments: HDFS-7224.v1.201410301923.txt
In very large clusters, the webhdfs client could get bind exceptions because
it runs out of ephemeral
ports. This could happen when using webhdfs to talk to the NN in order to do
list globbing of a
huge amount of files.
WebHdfsFileSystem#jsonParse gets the input/error stream from the connection,
but never closes the stream. Since it's not closed, the JVM thinks the stream
may still
be transferring data, so the next time through this code, it has to get a new
connection
rather than reusing an existing one.
The lack of connection reuse has poor latency and adds too much overhead to
the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
