I can take a look later tonight -- feel free to do it sooner if you'd
like to rule it out as a cause of the leak & debug that.
On Mon, Jun 30, 2008 at 2:56 PM, Oleg Kalnichevski <[EMAIL PROTECTED]> wrote:
> On Mon, 2008-06-30 at 14:54 -0400, Sam Berlin wrote:
>> Yeah, that makes sense. Dunno what was going through my head at the
>> time, but for some reason I thought it made sense to calculate the
>> time when the headers are read. It definitely simplifies the code to
>> do it just when the connection is released.
>>
>> Sam
>>
>
> Can you take care of that or shall I do it?
>
> Oleg
>
>
>> On Mon, Jun 30, 2008 at 2:51 PM, Oleg Kalnichevski <[EMAIL PROTECTED]> wrote:
>> > On Mon, 2008-06-30 at 10:43 -0400, Sam Berlin wrote:
>> >> Yikes, that's no good. Did you test this using the first revision of
>> >> the fix or the second (changed to use 'setIdleDuration' in
>> >> ManagedClientConnection)?
>> >>
>> >> Think it'd be possible to isolate the problem to a unit test?
>> >>
>> >> Sam
>> >>
>> >
>> > Sam,
>> >
>> > Maybe unrelated to the resource leak but something that struck me as
>> > odd. Currently AbstractClientConnAdapter starts counting the idle time
>> > from the moment of receiving the last header. This does not seem right
>> > to me. I guess we should start counting from the moment of the last I/O
>> > on that connection. For simplicity reasons I think we can assume this to
>> > be the moment the connection is released back to the manager, as
>> > connections get released automatically immediately when the data stream
>> > returns EOF.
>> >
>> > Does this make sense?
>> >
>> > Oleg
>> >
>> >
>> >> On Mon, Jun 30, 2008 at 10:33 AM, Paul Lindner <[EMAIL PROTECTED]> wrote:
>> >> >
>> >> > I tested this code using production traffic and the results are not so
>> >> > good.
>> >> > Ended up with a connection/memory leak. The jmap histogram of a
>> >> > server in
>> >> > this bad state is attached.
>> >> >
>> >> > Also, I had some issues with one host, it used a keepalive with a
>> >> > timeout=1.
>> >> > It appears that the route pool filled up with connections. It also
>> >> > appears
>> >> > that when it closes an expired free connection that it is not released
>> >> > back
>> >> > into the pool... Here's the debug output showing the final connection
>> >> > allocation and the attempt to get a connection from a full pool. It
>> >> > appears
>> >> > that Freeing the connection puts it back on the Idle pool, but it
>> >> > becomes
>> >> > inaccessible for some reason.
>> >> >
>> >> >
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager -
>> >> > ThreadSafeClientConnManager.getConnection:
>> >> > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout =
>> >> > 50002008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free
>> >> > connection
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free
>> >> > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.DefaultClientConnection - Connection closed
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 1
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Creating new
>> >> > connection
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]]
>> >> > 2008-06-29 19:22:27,761 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.DefaultClientConnection - Connection closed
>> >> > 2008-06-29 19:22:27,818 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:27,818 [http-80-55] DEBUG
>> >> > org.apache.http.impl.conn.IdleConnectionHandler - Adding connection at:
>> >> > 12147925478182008-06-29 19:22:27,818
>> >> > [http-80-55] DEBUG org.apache.http.impl.conn.tsccm.ConnPoolByRoute -
>> >> > Notifying no-one, there are no waiting threads
>> >> >
>> >> > -----------------------------------------------
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager -
>> >> > ThreadSafeClientConnManager.getConnection:
>> >> > HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com], timeout = 5000
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Getting free
>> >> > connection
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Closing expired free
>> >> > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.DefaultClientConnection - Connection closed
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,274 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for
>> >> > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>> >> > org.apache.http.impl.conn.DefaultClientConnection - Connection shut
>> >> > down
>> >> > 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Freeing connection
>> >> > [HttpRoute[{}->http://ads.socialmedia.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-79] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Notifying thread
>> >> > waiting
>> >> > on any pool
>> >> > 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for
>> >> > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - No free connections
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Available capacity: 0
>> >> > [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> > 2008-06-29 19:22:29,331 [http-80-26] DEBUG
>> >> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute - Need to wait for
>> >> > connection [HttpRoute[{}->http://ayi.h5.snapinteractiveapps.com]][null]
>> >> >
>> >> >
>> >> >
>> >> > Size Count Class description
>> >> > -------------------------------------------------------
>> >> > 684164112 558913 byte[]
>> >> > 492584016 2134710 char[]
>> >> > 209551728 4365661
>> >> > java.util.concurrent.locks.ReentrantLock$NonfairSync
>> >> > 209517696 4364952 java.util.concurrent.ConcurrentHashMap$Segment
>> >> > 144470200 4364956
>> >> > java.util.concurrent.ConcurrentHashMap$HashEntry[]
>> >> > 101379880 2534497 java.lang.String
>> >> > 65693112 628849 java.lang.Object[]
>> >> > 47067200 588340 com.friend.data.UserAppBean
>> >> > 41467120 272810 java.util.concurrent.ConcurrentHashMap$Segment[]
>> >> > 37081896 272661 org.apache.catalina.session.StandardSession
>> >> > 32830608 683971 java.util.HashMap$Entry
>> >> > 30965200 275343 java.util.Hashtable$Entry[]
>> >> > 24135760 603394 java.util.ArrayList
>> >> > 22612816 115641 java.util.HashMap$Entry[]
>> >> > 19871936 620998 java.util.Date
>> >> > 19642320 272810 java.util.concurrent.ConcurrentHashMap
>> >> > 18644208 224156 int[]
>> >> > 17608256 275129 java.util.Hashtable
>> >> > 14569296 303527 java.util.concurrent.ConcurrentHashMap$HashEntry
>> >> > 13089600 272700 java.beans.PropertyChangeSupport
>> >> > 13054072 97229 * ConstMethodKlass
>> >> > 12340480 154256 org.apache.tomcat.util.buf.MessageBytes
>> >> > 11679960 97229 * MethodKlass
>> >> > 10593856 165529 org.apache.tomcat.util.buf.ByteChunk
>> >> > 10529104 10652 * ConstantPoolKlass
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: [EMAIL PROTECTED]
>> >> > For additional commands, e-mail: [EMAIL PROTECTED]
>> >> >
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> >> For additional commands, e-mail: [EMAIL PROTECTED]
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [EMAIL PROTECTED]
>> > For additional commands, e-mail: [EMAIL PROTECTED]
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
