[ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12432679 ] Doug Cutting commented on HADOOP-312: -------------------------------------
You're right, Client.running is inapproprate for this use, and the culler does in fact remove connections (not sure how I missed that). Re modes: right now there are three modes, each with some separate logic: full-caching, timed caching, and no caching. It would simplify the logic (and hence ease testing and increase reliability) if there were only one mode. Otherwise, we should add unit tests for all modes. But the timed cache seems fairly universal. Setting the timeout to Integer.MAX_VALUE gives full-caching, while setting it to zero could cause most connections to terminate after a single request (although, if some get reused, that should not cause problems). The default should be a fairly short timeout, so that some kinds of requests will reuse cached connections and some will create new connections per request. This way the default configuration will test all of the code. I do not like having modes that might be useful but are not regularly used, since they'll not be tested and may stop working and then generate bugs. > Are you saying that we should remove "if (isCachingDisabled)" and instead > have the corresponding logic work under "if (maxIdleTime == 0)" ? I don't think we should have either. MaxIdleTime==0 is simply a very short idle time. The culler thread should have a minimum sleep time so that it doesn't busywait. > Connections should not be cached > -------------------------------- > > Key: HADOOP-312 > URL: http://issues.apache.org/jira/browse/HADOOP-312 > Project: Hadoop > Issue Type: Improvement > Components: ipc > Reporter: Devaraj Das > Assigned To: Devaraj Das > Attachments: no_conn_caching.patch, no_conn_caching.patch, > no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch, > no_conn_caching.patch, no_connection_caching.patch, > no_connection_caching.patch > > > Servers and clients (client include datanodes, tasktrackers, DFSClients & > tasks) should not cache connections or maybe cache them for very short > periods of time. Clients should set up & tear down connections to the servers > everytime they need to contact the servers (including the heartbeats). If > connection is cached, then reuse the existing connection for a few subsequent > transactions until the connection expires. The heartbeat interval should be > more so that many more clients (order of tens of thousands) can be > accomodated within 1 heartbeat interval. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
