[ https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16731542#comment-16731542 ]
Subham Mittal edited comment on HDFS-8855 at 1/1/19 8:45 AM: ------------------------------------------------------------- Check [https://javahungry.blogspot.com/2013/08/singleton-design-pattern-use-in-java.html] for a nice detail about idiomatic singletons in Java. was (Author: harinder07): SeeĀ [https://javahungry.blogspot.com/2013/08/singleton-design-pattern-use-in-java.html] for a nice little discussion of idiomatic singletons in Java > Webhdfs client leaks active NameNode connections > ------------------------------------------------ > > Key: HDFS-8855 > URL: https://issues.apache.org/jira/browse/HDFS-8855 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs > Reporter: Bob Hansen > Assignee: Xiaobing Zhou > Priority: Major > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-8855.005.patch, HDFS-8855.006.patch, > HDFS-8855.007.patch, HDFS-8855.008.patch, HDFS-8855.009.patch, > HDFS-8855.1.patch, HDFS-8855.2.patch, HDFS-8855.3.patch, HDFS-8855.4.patch, > HDFS_8855.prototype.patch > > > The attached script simulates a process opening ~50 files via webhdfs and > performing random reads. Note that there are at most 50 concurrent reads, > and all webhdfs sessions are kept open. Each read is ~64k at a random > position. > The script periodically (once per second) shells into the NameNode and > produces a summary of the socket states. For my test cluster with 5 nodes, > it took ~30 seconds for the NameNode to have ~25000 active connections and > fails. > It appears that each request to the webhdfs client is opening a new > connection to the NameNode and keeping it open after the request is complete. > If the process continues to run, eventually (~30-60 seconds), all of the > open connections are closed and the NameNode recovers. > This smells like SoftReference reaping. Are we using SoftReferences in the > webhdfs client to cache NameNode connections but never re-using them? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org