Re: Memory Leaks when web server hangs
Hi Eric, Reusing the MultiThreadedHttpConnectionManager is definitely the way to go. As Roland mentions reusing HttpClient instances might help performance, but it is not necessary. Mike On Mar 22, 2004, at 2:01 AM, Eric Bloch wrote: I'm happy to change my code but have a version of my product out there that does it this way and wanted to make sure it wasn't in need of immediate patch :-) -E Roland Weber wrote: Hello Eric, by using the same connection manager for each HttpClient, you avoid the biggest penalty for creating new clients. But still, each new client uses up some heap space, which will have to be garbage collected sooner or later. There are state objects, and parameter objecs, and probably some more that will be created. Nothing that hurts much, but what gain do you expect from creating new HttpClient objects? You don't have to. cheers, Roland Eric Bloch [EMAIL PROTECTED] 22.03.2004 07:03 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc:Subject:Re: Memory Leaks when web server hangs Oh... and I just want to know if that's not the httpclient way, too? That is, should I be reusing the client? What will that get me if I'm already reusing the connection manager? Thanks again, Eric Eric Bloch wrote: I am creating one MultiThreadedHttpConnectionManager and creating/destroying HttpClient each time but always constructing with the same connection manager. Thanks, Eric Michael Becke wrote: Hi Eric, What exactly do you mean by thread thrashing? Which connection manager are you using? Mike On Mar 19, 2004, at 1:21 PM, Eric Bloch wrote: Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc:Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish
Re: Memory Leaks when web server hangs
Hi Eric, What exactly do you mean by thread thrashing? Which connection manager are you using? Mike On Mar 19, 2004, at 1:21 PM, Eric Bloch wrote: Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc:Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/ threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park
Re: Memory Leaks when web server hangs
Hello Eric, by using the same connection manager for each HttpClient, you avoid the biggest penalty for creating new clients. But still, each new client uses up some heap space, which will have to be garbage collected sooner or later. There are state objects, and parameter objecs, and probably some more that will be created. Nothing that hurts much, but what gain do you expect from creating new HttpClient objects? You don't have to. cheers, Roland Eric Bloch [EMAIL PROTECTED] 22.03.2004 07:03 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: Memory Leaks when web server hangs Oh... and I just want to know if that's not the httpclient way, too? That is, should I be reusing the client? What will that get me if I'm already reusing the connection manager? Thanks again, Eric Eric Bloch wrote: I am creating one MultiThreadedHttpConnectionManager and creating/destroying HttpClient each time but always constructing with the same connection manager. Thanks, Eric Michael Becke wrote: Hi Eric, What exactly do you mean by thread thrashing? Which connection manager are you using? Mike On Mar 19, 2004, at 1:21 PM, Eric Bloch wrote: Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc:Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/ threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use
Re: Memory Leaks when web server hangs
I'm happy to change my code but have a version of my product out there that does it this way and wanted to make sure it wasn't in need of immediate patch :-) -E Roland Weber wrote: Hello Eric, by using the same connection manager for each HttpClient, you avoid the biggest penalty for creating new clients. But still, each new client uses up some heap space, which will have to be garbage collected sooner or later. There are state objects, and parameter objecs, and probably some more that will be created. Nothing that hurts much, but what gain do you expect from creating new HttpClient objects? You don't have to. cheers, Roland Eric Bloch [EMAIL PROTECTED] 22.03.2004 07:03 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: Memory Leaks when web server hangs Oh... and I just want to know if that's not the httpclient way, too? That is, should I be reusing the client? What will that get me if I'm already reusing the connection manager? Thanks again, Eric Eric Bloch wrote: I am creating one MultiThreadedHttpConnectionManager and creating/destroying HttpClient each time but always constructing with the same connection manager. Thanks, Eric Michael Becke wrote: Hi Eric, What exactly do you mean by thread thrashing? Which connection manager are you using? Mike On Mar 19, 2004, at 1:21 PM, Eric Bloch wrote: Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc:Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning
Re: Memory Leaks when web server hangs
Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park West 4121 Australia QLD
Re: Memory Leaks when web server hangs
On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park West 4121 Australia QLD www.intencha.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory Leaks when web server hangs
Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park West 4121 Australia QLD www.intencha.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory Leaks when web server hangs
Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park West 4121 Australia QLD www.intencha.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Srinivas Vemula +91 40 23547826- Ext 201 Associate Consultant +91 40 23541447 (Fax) Mensamind+91 98497-42720 (Mobile) Hyderabad India http://www.mensamind.com DISCLAIMER The information contained in this e-mail is confidential and intended for the named recipient(s) only. If you are not an intended recipient of this email you must not copy, distribute or take any further action in reliance on it. You should delete it and notify the sender immediately.
Re: Memory Leaks when web server hangs
Hi Srinivas, et al, Hope all is well. We've also experienced a similar story here in Oz just the other day. Essentially connections to the webserver were not being released and were consuming sockets on the client (determined via OptimizeIT). After searching the dev list I found a post by another developer that said this was the default behaviour for HTTP 1.1 which HttpClient honoured (ie. keep-alive), if the server didn't indicate what the client should do with the connection. When I looked into the headers via a wire trace I saw that indeed the server (IIS in this case) didn't send a 'Connection: close' header and after browsing the HttpClient source saw that the connection was kept alive if using HTTP 1.1. The non-programatic way to fix it was to add a request header to each client request saying 'Connection: close'. After doing this, HttpClient closed all connections after they were released and the open socket problem went away. Hope this helps. Cheers, Marcus On Mon, Feb 02, 2004 at 04:26:39PM +0530, Srinivas Vemula wrote: Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian
Re: Memory Leaks when web server hangs
Marcus Crafter wrote: The non-programatic way to fix it was to add a request header to each client request saying 'Connection: close'. You could also have written your own connection manager implementation instead of messing with request headers. This would have been more portable to later versions. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory Leaks when web server hangs
Hi Srinivas, Yes, that's what we did here, afterwards the connections that were previously held open, were closed. Sorry if it wasn't clearer in my first email. Cheers, Marcus On Mon, Feb 02, 2004 at 05:31:33PM +0530, Srinivas Vemula wrote: Thanks Marcus, Are u saying we should add getMethod.addRequestHeader(Connection, close) for every request made using HttpClient ? Excuse me if I missed your point completely. srini Marcus Crafter wrote: Hi Srinivas, et al, Hope all is well. We've also experienced a similar story here in Oz just the other day. Essentially connections to the webserver were not being released and were consuming sockets on the client (determined via OptimizeIT). After searching the dev list I found a post by another developer that said this was the default behaviour for HTTP 1.1 which HttpClient honoured (ie. keep-alive), if the server didn't indicate what the client should do with the connection. When I looked into the headers via a wire trace I saw that indeed the server (IIS in this case) didn't send a 'Connection: close' header and after browsing the HttpClient source saw that the connection was kept alive if using HTTP 1.1. The non-programatic way to fix it was to add a request header to each client request saying 'Connection: close'. After doing this, HttpClient closed all connections after they were released and the open socket problem went away. Hope this helps. Cheers, Marcus On Mon, Feb 02, 2004 at 04:26:39PM +0530, Srinivas Vemula wrote: Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default
RE: Memory Leaks when web server hangs
Srini, If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. As any other java application HttpClient is not in a position to dispose of any unused objects. This is the job of Java's garbage collection. Java application may only politely ask GC to clean up but there's no way of reliably forcing garbage collection. This is how connect timeout is implemented (Odi, please correct me if I miss something): * HttpClient attempts to open a socket * if connect timeout is set to a positive value, a controller thread is started. * the controller thread sleeps for the given number of milliseconds * if the controller thread detects that there's still no connection upon after being awoken up, if simply nulls the reference to the socket object awaiting connection, and lets GC to eventually dispose of that object * If GC keeps those objects alive for considerably long period of time, JVM may eventually run out of resources Ways to mend the situation (1) As Roland said please strongdo reuse/strong the HttpClient instance. HttpClient class is fully thread-safe. By no reusing HttpClient you completely defeat the connection keep-alive stuff (2) Use the trick suggested by Marcus (3) If you do not want to lose the benefits of connection reuse you have two options: * do not use HttpClient#setConnectionTimeout and pray that default JVM connect timeout setting is sane * fork HttpClient and make HttpConnection use Socket#connect(SocketAddress, int) (available as of java 1.4) instead of older Socket#connect(InetAddress, int) + controller thread workaround http://java.sun.com/j2se/1.4.2/docs/api/java/net/Socket.html#connect(java.net.SocketAddress,%20int) Oleg -Original Message- From: Srinivas Vemula [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 11:57 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want
Re: Memory Leaks when web server hangs
Hi Srini, I want to echo Oleg's comments with one addition. (1) As Roland said please strongdo reuse/strong the HttpClient instance. HttpClient class is fully thread-safe. By no reusing HttpClient you completely defeat the connection keep-alive stuff HttpClient is thread safe, depending which connection manager is used. By default the SimpleHttpConnectionManager is used by HttpClient. This connection manager is not thread safe, though it can be replaced with the MultiThreadedHttpConnectionManager which is. Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
