Whew! Thanks for the heads-up about the problem, and confirmation
that the fix works, Paul.
Sam
On Tue, Jul 1, 2008 at 7:16 PM, Paul Lindner <[EMAIL PROTECTED]> wrote:
> That did it. I'm running svn head on one production server and seeing very
> good behavior.
>
> On Jun 30, 2008, at 6:53 PM, Sam Berlin wrote:
>
>> Hi Paul,
>>
>> I think I've fixed this one -- the problem was using deleteEntry
>> instead of dropEntry. The former required the entry to be considered
>> 'free', but the entry had just been allocated and was definitely not
>> 'free'. I'm pretty certain this will indirectly fix the
>> connection/memory leak -- a side effect of the problem was connection
>> requests would just pile up and never be serviced.
>>
>> Sam
>>
>> On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <[EMAIL PROTECTED]> wrote:
>>>
>>> I tested this code using production traffic and the results are not so
>>> good.
>>> Ended up with a connection/memory leak. The jmap histogram of a server
>>> in
>>> this bad state is attached.
>>>
>>> Also, I had some issues with one host, it used a keepalive with a
>>> timeout=1.
>>> It appears that the route pool filled up with connections. It also
>>> appears
>>> that when it closes an expired free connection that it is not released
>>> back
>>> into the pool... Here's the debug output showing the final connection
>>> allocation and the attempt to get a connection from a full pool. It
>>> appears
>>> that Freeing the connection puts it back on the Idle pool, but it becomes
>>> inaccessible for some reason.
>>>
>>>
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager -
>>> ThreadSafeClientConnManager.getConnection:
>>> HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout =
>>> 50002008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free
>>> connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection - Connection closed
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 1
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Creating new
>>> connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]]
>>> 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection - Connection closed
>>> 2008-06-29 19:22:27,818 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:27,818 [http-80-55] DEBUG
>>> org.apache.http.impl.conn.IdleConnectionHandler - Adding connection at:
>>> 12147925478182008-06-29 19:22:27,818
>>> [http-80-55] DEBUG org.apache.http.impl.conn.tsccm.ConnPoolByRoute -
>>> Notifying no-one, there are no waiting threads
>>>
>>> -----------------------------------------------
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager -
>>> ThreadSafeClientConnManager.getConnection:
>>> HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = 5000
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free
>>> connection
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection - Connection closed
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>>> org.apache.http.impl.conn.DefaultClientConnection - Connection shut down
>>> 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection
>>> [HttpRoute[{}->http://ads.socialmedia.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Notifying thread
>>> waiting
>>> on any pool
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0
>>> [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>> 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for
>>> connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>>>
>>>
>>>
>>> Size Count Class description
>>> -------------------------------------------------------
>>> 684164112 558913 byte[]
>>> 492584016 2134710 char[]
>>> 209551728 4365661
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync
>>> 209517696 4364952 java.util.concurrent.ConcurrentHashMap$Segment
>>> 144470200 4364956
>>> java.util.concurrent.ConcurrentHashMap$HashEntry[]
>>> 101379880 2534497 java.lang.String
>>> 65693112 628849 java.lang.Object[]
>>> 47067200 588340 com.friend.data.UserAppBean
>>> 41467120 272810 java.util.concurrent.ConcurrentHashMap$Segment[]
>>> 37081896 272661 org.apache.catalina.session.StandardSession
>>> 32830608 683971 java.util.HashMap$Entry
>>> 30965200 275343 java.util.Hashtable$Entry[]
>>> 24135760 603394 java.util.ArrayList
>>> 22612816 115641 java.util.HashMap$Entry[]
>>> 19871936 620998 java.util.Date
>>> 19642320 272810 java.util.concurrent.ConcurrentHashMap
>>> 18644208 224156 int[]
>>> 17608256 275129 java.util.Hashtable
>>> 14569296 303527 java.util.concurrent.ConcurrentHashMap$HashEntry
>>> 13089600 272700 java.beans.PropertyChangeSupport
>>> 13054072 97229 * ConstMethodKlass
>>> 12340480 154256 org.apache.tomcat.util.buf.MessageBytes
>>> 11679960 97229 * MethodKlass
>>> 10593856 165529 org.apache.tomcat.util.buf.ByteChunk
>>> 10529104 10652 * ConstantPoolKlass
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>
> Paul Lindner
> [EMAIL PROTECTED]
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
