re: ant file for converting src from commons-logging to log4j for 2.0.1
Looks like my attachment didn't come through. Did anyone see it? My sent box looks ok, but the message sent out via the mailing list is empty! -Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
ant file for converting src from commons-logging to log4j for 2.0.1
Attached is an ant build file that can be plopped into your basedir for
httpclient. (I tested it with the 2.0.1 src distro rather than a cvs
dir.)
% ant -buildfile commons-logging-to-log4j.xml -projecthelp
Buildfile: commons-logging-to-log4j.xml
Main targets:
convert convert source from commons-logging to
log4j apis
install-log4j-propertiescopy log4j properties to classes dir
make-src-backup make src.orig directory
revert Put original java source code back
uninstall-log4j-properties remove log4j properties from classes dir
With some massaging, these targets could probably be renamed and merged
into build.xml and maintained there (and probably belong there). Also,
in my envirionment, I used a build.properties that has
${commons-logging.jar} pointing to a copy of log4j.jar. If I were going
to maintain this inside the commons-httpclient/build.*, I'd rename the
commons-logging.jar property to ${logging-jar}, too, so things would
make a little more sense.
I ran the tests and it seems to basically work. I can now, happily say
'bubbye' to commons-logging.
Enjoy
-Eric
!--
Ant build file to convert httpclient java source code from commons-logging api
to direct use of log4j. Known to work with httpclient 2.0.1 src base.
--
project name=httpclient-commons-logging-to-log4j default=convert
basedir=.
target name=make-src-backup description=make src.orig directory
mkdir dir=src.orig /
copy todir=src.orig
fileset dir=src /
/copy
/target
target name=convert depends=make-src-backup
description=convert source from commons-logging to log4j apis
replace dir=src
replacefilter
token=org.apache.commons.logging.LogFactory
value=org.apache.log4j.Logger /
replacefilter
token=org.apache.commons.logging.Log
value=org.apache.log4j.Level /
replacefilter
token=LogFactory.getLog
value=Logger.getLogger /
replacefilter
token=LOG.trace
value=LOG.debug /
replacefilter
token=isTraceEnabled
value=isDebugEnabled /
replacefilter
token=isErrorEnabled()
value=isEnabledFor(Level.ERROR) /
replacefilter
token=isWarnEnabled()
value=isEnabledFor(Level.WARN) /
replacefilter
token=Log
value=Logger /
/replace
/target
!-- Useful while running tests --
target name=install-log4j-properties description=copy log4j properties to classes dir
copy file=src/conf/log4j.properties.sample
tofile=target/classes/log4j.properties/
/target
!-- But we don't want to ship this file so remove it --
target name=uninstall-log4j-properties
description=remove log4j properties from classes dir
delete file=target/classes/log4j.properties/
/target
target name=revert description=Put original java source code back
move todir=src
fileset dir=src.orig /
/move
/target
/project
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Problems with commons-logging jar and request for advice.
Thanks Oleg, Appreciate your bunch of random thoughts. We'll probably build a source code converter utility here and send it along once we're happy with it. Fwiw, your luck (or our lack of luck) may have to do with the fact that we use log4j natively; it's possible and plausible that mixed use of native log4j and commons-logging is what is causing us grief with certain containers. We're reasonably tucked into log4j that I can't see ditching it for commons-logging either. I thought logging was spose to help debugging not cause it! -Eric Oleg Kalnichevski wrote: Hi Eric Personally up to now I have not had too much of a hard time using commons-logging with Websphere 5.0 and 5.1. Maybe I was just lucky You may want to fork off the HttpClient 2.0 branch and host it on the SourceForge, for instance. HttpClient 2.0 is stable and will not be actively developed in the future. So you should not have too much trouble keeping the forked version in sync with the official one. The drawback of this approach is that once HttpClient 3.0 is out you may have to repeat the whole exercise again. So, probably a better solution would be to develop a simple converter, which would walk through the source code and replace all the references to commons-logging with log4j or jdk14. Such a converter could be used against any other commons-logging dependent library or application, so it might be a worthwhile investment. We'd happily include this utility into the contrib package Just a bunch of random thoughts Oleg On Sat, 2004-08-28 at 00:39, Eric Bloch wrote: Hey Folks, I've experienced a few problems with httpclient not instantiating in a web application under some containers (websphere 5.1 is the latest, but I've seen problems in tomcat and ATG as well). This turns out to *always* be a problem with class-loading and commons-logging. It's hard to precisely describe the problems, but it always seems like it has something to do with different class-loaders loading the commons-logging api or implementation jar. I'm wondering what the current advice is on common-logging. See http://www.qos.ch/logging/thinkAgain.html for details on links to the numerous problems and problem reports with commons-logging. I imagine I could be walking into a religious debate here, bu, as far as I can tell, commons-logging is basically broken wrt to its class-loader and the servlet-container spec for class-loading. (It always chooses the Java spec rather than the servlet container spec). I really only care about httpclient, but unfortunately, it seems I'm stuck with commons-logging because httpclient uses it. The only plan I can think of now is to remove commons-logging from httpclient. 1) Is there anyone else interested in a copy of httpclient modified to use either jdk1.4 logging or log4j logging directly? Any preferences (me I prefer log4j mostly because it's what I'm accustomed to). 2) Anyone have any advice on how to maintain a copy of httpclient that avoided commons-logging? 3) Anyone know any commons-logging folks I can email/talk to? FWIW, I'm not a class-loader expert. I've tried to explain the problems to commons-logging folks in a bug I filed (and in other bugs I've read), but I don't see this getting resolved in a timely fashion. -Eric A few details btw: To get things working in ATG, I had to: - unjar DAS/lib/classes.jar - remove org/apache/common/* - jar it back up into classes.jar I also removed the common_logging.jar file (it looks like there are some additional org.apache.common.logging classes in there too) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Problems with commons-logging jar and request for advice.
Hey Folks, I've experienced a few problems with httpclient not instantiating in a web application under some containers (websphere 5.1 is the latest, but I've seen problems in tomcat and ATG as well). This turns out to *always* be a problem with class-loading and commons-logging. It's hard to precisely describe the problems, but it always seems like it has something to do with different class-loaders loading the commons-logging api or implementation jar. I'm wondering what the current advice is on common-logging. See http://www.qos.ch/logging/thinkAgain.html for details on links to the numerous problems and problem reports with commons-logging. I imagine I could be walking into a religious debate here, bu, as far as I can tell, commons-logging is basically broken wrt to its class-loader and the servlet-container spec for class-loading. (It always chooses the Java spec rather than the servlet container spec). I really only care about httpclient, but unfortunately, it seems I'm stuck with commons-logging because httpclient uses it. The only plan I can think of now is to remove commons-logging from httpclient. 1) Is there anyone else interested in a copy of httpclient modified to use either jdk1.4 logging or log4j logging directly? Any preferences (me I prefer log4j mostly because it's what I'm accustomed to). 2) Anyone have any advice on how to maintain a copy of httpclient that avoided commons-logging? 3) Anyone know any commons-logging folks I can email/talk to? FWIW, I'm not a class-loader expert. I've tried to explain the problems to commons-logging folks in a bug I filed (and in other bugs I've read), but I don't see this getting resolved in a timely fashion. -Eric A few details btw: To get things working in ATG, I had to: - unjar DAS/lib/classes.jar - remove org/apache/common/* - jar it back up into classes.jar I also removed the common_logging.jar file (it looks like there are some additional org.apache.common.logging classes in there too) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
running with a policy file and system properties
I'm walking down the task of making a product that uses httpclient run underneath a server with a java policy file. I know httpclient makes threads and sockets and I believe I have these under control (will be allowed or disabled), but I've seen some code in the 2.0 code base that does System.getProperties().getProperty(foo). Why would you do that instead of System.getProperty(foo) (which requires only read access to the property rather than full read/write access to all properties) ? In general, in a server with a policy file, you shouldn't ever depend on reading a property (because you might get a SecurityException). I need to either (1) catch the SecurityException or (2) use a simpler version of the code that can be enabled without giving access to all system properties. -Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Wire log to Log4J logger.
Hey Folks, I'm having trouble making httpclient 2.0 use log4j logging inside Tomcat5 (when running against java 1.4). Seems as though tomcat5 makes some call to the the commons logger early on (see below). Tomcat5 doesn't come with log4j... so the commons logging default log factory implementation picks up the jdk14 logger for tomcat as you'd expect. I'm guessing that somehow when my webapp comes up... the commons logger fails to look for log4j in my WEB-INF/lib. This feels like one of the class-loader disagreements that I've seen related to java/commons-logging and the servlet spec. That is, my webapp is getting tomcat's copy of the default log factor impl, instead of its own. Fwiw, this is a stack from early tomcat initialization: LogFactoryImpl.getLogClassName() line: 331 LogFactoryImpl.getLogConstructor() line: 368 LogFactoryImpl.newInstance(String) line: 529 LogFactoryImpl.getInstance(String) line: 235 LogFactoryImpl.getInstance(Class) line: 209 LogFactory.getLog(Class) line: 351 JdkCompat.clinit() line: 53 StandardClassLoader.clinit() line: 207 ClassLoaderFactory.createClassLoader(File[], File[], URL[], ClassLoader) line: 189 Bootstrap.createClassLoader(String, ClassLoader) line: 160 Bootstrap.initClassLoaders() line: 104 Bootstrap.init() line: 193 Bootstrap.main(String[]) line: 399 And this is a stack where my initial use of the httpclient ends up picking up the JDK14 logger even though my webapp has log4j in it's WEB-INF/lib: Jdk14Logger.init(String) line: 50 NativeConstructorAccessorImpl.newInstance0(Constructor, Object[]) line: not available [native method] NativeConstructorAccessorImpl.newInstance(Object[]) line: 39 DelegatingConstructorAccessorImpl.newInstance(Object[]) line: 27 Constructor.newInstance(Object[]) line: 274 LogFactoryImpl.newInstance(String) line: 529 LogFactoryImpl.getInstance(String) line: 235 LogFactoryImpl.getInstance(Class) line: 209 LogFactory.getLog(Class) line: 351 MultiThreadedHttpConnectionManager.clinit() line: 98 . . . It isn't an option for me to put touch any of the jars inside the container. And I'd like to force httpclient to use log4j when it's used within my webapp. This was working fine, fyi, in tomcat4. Any clues/advice? Thanks! -Eric Michael Becke wrote: Hi John, HttpClient uses commons-logging which defaults to Log4j when it is present on the classpath. You can enable wire logging by setting the logger httpclient.wire to DEBUG. This can be done using any of Log4j's standard configuration methods. Mike On May 24, 2004, at 10:07 AM, John Melody wrote: Is there a way to configure the wire log to go to a Log4J log configured for the application. I have configured Log4J as my application logger and I would like to send the Httpclient wire log to the same file. Currently it is set up as the following - System.setProperty(org.apache.commons.logging.simplelog.log.httpclient .wire , debug); which sends the wire log to the console. Is it possible to configure this so that all the logging output including the wire log would go to my log4J application log? thanks for any help, regards, John. John Melody SyberNet Ltd. Galway Business Park, Dangan, Galway. Tel. No. +353 91 514400 Fax. NO. +353 91 514409 Mobile - 087-2345847 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: questionable default value for BufferedOutputStream size in HttpConnection and memory usage?
JMX has not been formally considered or discussed on this list. I personally do not think it would have made a good match HttpClient needs to allow for customization of lots of short-lived objects (HttpMethods, HostConfigurations, HttpConnections, etc), whereas JMS is better suited for relatively few long-lived objects that usually represent some type of services (at least IHMO). JMX would be simply too much, too heavy-weight. For HttpClient's needs light-weight linked hash-maps do just fine. This said, an JMX layer sitting on top of HttpClient that provide MBeans for HttpClient instances would be a great contribution Hey Oleg, Next time I have the cycles to sit down with JMX *and* port Laszlo's product from httpclient 2.0 to httpclient 3.0, I may be able to get to this :-). I'd be focused on settings for a client using the MultiThreadedConnectionManager. Btw, how do I go about getting the Laszlo Presentation Server listed on the applications page for httpclient? We've been a user since 1.0 actually. We use httpclient as the core HTTP transport for a transcoding proxy inside the Laszlo Presentation Server. See http://www.laszlosystems.com/ for all sorts of details including free developer and non-commericial-use downloads and documentation. As to a short blurb for the 'applications page', something like: The Laszlo Presentation Server is an XML-native platform for the development and delivery of a new generation of Rich Internet Applications. For a quick introduction see: http://www.laszlosystems.com/lps/laszlo-in-ten-minutes/ . Btw, we also have an interest in using httpclient as the transport for a SOAP client if anyone else is working on that. Best, -Eric Bloch - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
questionable default value for BufferedOutputStream size in HttpConnection and memory usage?
Hi httpclient folks,
I've been looking at 2.0 source code and the default value for the
BufferedOutputStream that is used in an HttpConnectionn is coming from
socket.getSendBufferSize(). My hunch, is that, in general, this is
bigger than you'd want.
Most HTTP sends are less than 1KByte ('cept for big POSTs).
The default value I get for socket.getSendBufferSize for this is 8192.
I would think a better default for this buffer would be 1K, no?
Also, fyi, if someone happens to dork the system send buffer size hi
(say MB) and you are using the MultiThreadedConnectionManager in 2.0
(dunno about 3.0), you will use up a lot of memory for each connection
since the pool doesn't let idle connections (or their buffers) be gced.
I just got bit bad by that.
-Eric
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: questionable default value for BufferedOutputStream size in HttpConnection and memory usage?
Thanks, I filed against 2.0 final. A question: did you guys consider
jmx for your 'preferences architecture' ?
Thanks,
Eric
Oleg Kalnichevski wrote:
Hi Eric
Thanks for bringing this up. HttpClient 3.0 allows for parameterization
of SO_SNDBUF and SO_RCVBUF settings. For HttpClient 2.0 (as well as for
3.0 when falling back onto the system defaults), however, it would make
sense to set a cap on the size of the send and receive buffers.
Feel free to open a ticket for this issue with Bugzilla
Oleg
On Fri, 2004-07-02 at 18:39, Eric Bloch wrote:
Hi httpclient folks,
I've been looking at 2.0 source code and the default value for the
BufferedOutputStream that is used in an HttpConnectionn is coming from
socket.getSendBufferSize(). My hunch, is that, in general, this is
bigger than you'd want.
Most HTTP sends are less than 1KByte ('cept for big POSTs).
The default value I get for socket.getSendBufferSize for this is 8192.
I would think a better default for this buffer would be 1K, no?
Also, fyi, if someone happens to dork the system send buffer size hi
(say MB) and you are using the MultiThreadedConnectionManager in 2.0
(dunno about 3.0), you will use up a lot of memory for each connection
since the pool doesn't let idle connections (or their buffers) be gced.
I just got bit bad by that.
-Eric
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Laszlo Systems, Inc.
1040 Mariposa Street, SF, CA, USA 94107
voice: 415.241.2721
fax: 415.865.2914
im:eedeebee at yim; eedeebee3 at aim
email: [EMAIL PROTECTED]
--
Laszlo allows Behr to deliver a breakthrough experience with
ColorSmart by BEHR application at http://www.behr.com
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: unable to find line starting with HTTP
I am seeing this exact same problem with 2.0rc1 as well. A wget to the
server from the same client works fine. The httpclient wire log shows
that I get a redirect and then the in waiting for the HTTP status line
from the redirect, it never comes...
What server are you running?
-Eric
Kalnichevski, Oleg wrote:
Juan,
Usually HttpClient reports 'unable to find line starting with HTTP' when the target server drops the connection without returning any response. This can happen, for instance, if the server is being under heavy load.
(1) If you have access to the target server, examine the server logs to see why the
connection was dropped. Fixing the problem on the server side would be the best
(2) Another possible solution to this problem _may_ be to retry the request, provided that the
POST methods your code is supposed to execute are idempotent. If the server application, however,
has been designed in a way that it may receive a request, parse it, change the application state
(by committing some data to the data store, for instance) and then simply drop the connection
without giving any kind of response back to the client, you are in
strongBIG/strong trouble, as simply retrying the same POST method case cause data
inconsistency problems. So, the best thing to do is to get in touch with the server guys and find
out if it is safe to retry POST methods.
Hope this clarifies things a little
Oleg
-Original Message-
From: Juan Pedro López Sáez [mailto:[EMAIL PROTECTED]
Sent: Wednesday, June 02, 2004 17:18
To: [EMAIL PROTECTED]
Subject: unable to find line starting with HTTP
Hi,
I've been searching in the archive list looking for something related to
my unable to find line starting with HTTP.
There are lots of comments about it, but they all seem to be unuseful
for me.
Currently I'm using 2.0-rc3 version. My HTTP client is very simple.
Below you can see everything I do. That code is executed everytime I
need to send new data to the HTTP server.
--
HttpClient client= new HttpClient();
client.setStrictMode(true);
client.setTimeout(12);
client.setConnectionTimeout(1);
PostMethod post = new PostMethod(currentURL);
post.setRequestContentLength(XMLRequest.length());
post.setRequestBody(XMLRequest);
post.setRequestHeader(Content-type, text/xml; charset=ISO-8859-1);
post.addRequestHeader(Connection, close);
String response;
try{
httpStatus = client.executeMethod(post);
response = post.getResponseBodyAsString();
}
catch (HttpException e) {
log.fatal(IOException in client.executeMethod( post ),e);
throw e;
}
catch(IOException e){
log.fatal(IOException in client.executeMethod( post ),e);
throw e;
}
finally {
post.releaseConnection();
}
I'm getting the exception in the HttpClient.excetuteMethod line. The
exception happens from time to time, is not quite common but I don't
know what to do with it.
The exception stack trace is the following:
org.apache.commons.httpclient.HttpRecoverableException:
org.apache.commons.httpclient.HttpRecoverableException: Error in parsing
the status line from the response: unable to find line starting with
HTTP
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1965)
at
org.apache.commons.httpclient.HttpMethodBase.processRequest(HttpMethodBase.java:2659)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1093)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:674)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:529)
I would apreciate any help.
Thank you very much
Juan Pedro López
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory Leaks when web server hangs
I'm happy to change my code but have a version of my product out there that does it this way and wanted to make sure it wasn't in need of immediate patch :-) -E Roland Weber wrote: Hello Eric, by using the same connection manager for each HttpClient, you avoid the biggest penalty for creating new clients. But still, each new client uses up some heap space, which will have to be garbage collected sooner or later. There are state objects, and parameter objecs, and probably some more that will be created. Nothing that hurts much, but what gain do you expect from creating new HttpClient objects? You don't have to. cheers, Roland Eric Bloch [EMAIL PROTECTED] 22.03.2004 07:03 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: Memory Leaks when web server hangs Oh... and I just want to know if that's not the httpclient way, too? That is, should I be reusing the client? What will that get me if I'm already reusing the connection manager? Thanks again, Eric Eric Bloch wrote: I am creating one MultiThreadedHttpConnectionManager and creating/destroying HttpClient each time but always constructing with the same connection manager. Thanks, Eric Michael Becke wrote: Hi Eric, What exactly do you mean by thread thrashing? Which connection manager are you using? Mike On Mar 19, 2004, at 1:21 PM, Eric Bloch wrote: Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc:Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning
Re: Memory Leaks when web server hangs
Hey there, I create/destroy http clients but always have them use the same connection manager. Will that cause thread thrashing? Thanks, -Eric Roland Weber wrote: Hello Srini, you should *not* create a new HTTP Client for each request! This will also create a new connection manager, and a new cleanup thread. It beats the whole purpose of connection management. You should create *one* HTTP Client object for the lifetime of your application. Then, the connection manager of that client will reuse and free connections as appropriate. best regards, Roland Srinivas Vemula [EMAIL PROTECTED] 02.02.2004 11:56 Please respond to Commons HttpClient Project To: Commons HttpClient Project [EMAIL PROTECTED] cc: Subject:Re: Memory Leaks when web server hangs Oleg, We are using JDK1.4.1 and connection time out is set to 30Ms and read time out to 60Ms. We are trying to connect to a IP Camera, and send a heart beat command to the camera to check for its availability every 30 Secs from the time we find it is un available. If the web server is down, we feel that HttpClient is not cleaning up failed connections properly and thats resulting in Memory Leaks at the server side. The code we use is the standard way of using HttpClient and all the code is in a method and new HttpClient object is created for every request. Are there any ways to make sure the connections are all being purged properly? Are there any precautionary measures or flags we can set on HttpClient API when communicating with a web server (running on firm ware) with relatively less RAM and processing power (IpCamera with built in web server)? Thanks for your time and help. Srini Kalnichevski, Oleg wrote: Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Hi Srini, HttpClient uses an additional controller thread to work around the limitation of older ( 1.4) JDKs which do not provide a possibility to set connect timeout. If you do not really need to control connect timeout (for instance, when communicating with an intranet site with good availability) simply set connect timeout to. That will prevent HttpClient from spawning an additional thread per request. Adrian, et al Another possibility to use reflection to set connect timeout using the Socket methods when running in JVM 1.4 or above Oleg -Original Message- From: Adrian Sutton [mailto:[EMAIL PROTECTED] Sent: Monday, February 02, 2004 09:03 To: Commons HttpClient Project Subject: Re: Memory Leaks when web server hangs On 2/2/04 2:00 PM, Srinivas Vemula [EMAIL PROTECTED] wrote: Hi All, We are seeing thread leaks when having client open connections to a web server that hangs. Has any one seen this happening?? How do we ensure that the library correctly closes socket connections on failures, cleaning up system resources, and threads actually finish in the timeout period and get freed up. Would using MultiThreadedHttpConnectionManager file:///D:/silkroad/http-commons/commons-httpclient-2.0-rc3/docs/threading.ht ml#MultiThreadedHttpConnectionManager be of any help?? Hi Srini, If you're using the same HttpClient instance across multiple threads, you must use the MultiThreadedHttpConnectionManager or you'll run into strange problems. If you're using separate HttpClient instances, you may as well stick with the single threaded (default) connection manager. In terms of connections hanging - you probably want to look into the setConnectionTimeout and setTimeout methods of the HttpClient class. These allow you to control how long HttpClient waits when making a connection and how long it waits for data once the connection is established. If you have set either of these to 0 (not sure what the default is, it may be platform specific) the connection will never timeout which sounds a lot like what you're seeing. Mike and Oleg, the stack trace Srini included indicates that HttpClient is attempting to create a new timeout thread on every connection - does this always occur or is it just one timeout thread per httpclient instance? It would be ideal if we could reduce this to one static thread as starting threads is never nice for server side apps. Not sure how feasible that is though. Srini Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park West 4121 Australia QLD
Re: 2.0 rc1 leaking sockets?
Michael Becke wrote: Hi Eric, By default 20 is max number of total connections. This number will not be exceeded regardless of the number of hosts being connected to. What if maxconnsperhost is maxtotal ? For example, imagine I'm setting maxconnsperhost to 1000. Thanks, Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
question about SimpleHttpConnectionManager
I feel like this has been asked before I'm wondering how I force a socket to close when I'm using this manager. I couldn't find the api. Thanks, Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: 2.0 rc1 leaking sockets?
Thanks... just to be clear though... the multi-threadded connection mgr should never hold open more than a fixed number of sockets though, right? I've got a case where there's a clear socket leak in the java process and someone is pointing at me right now (and I use httpclient and always release my connections). -Eric Michael Becke wrote: Hi Eric, The multi-threaded connection manager holds on to connections that have been released, but it does not necessarily close them. Connections are only closed if the server sends a connection: close header or if HTTP 1.0 is used. There are currently two open bugs that may apply to your situation: http://issues.apache.org/bugzilla/show_bug.cgi?id=25372 and http://issues.apache.org/bugzilla/show_bug.cgi?id=27589 I would also suggest checking the archive for more discussion on this matter. Mike On Mar 16, 2004, at 7:31 PM, Eric Bloch wrote: Hi there, I've got a question. I'm using the Multithreaded Connection Manager and I know that I'm always calling releaseConnection() on every connection I use. Are there any reports (or possibilities) that releaseConnection() will fail to close an open socket or lose track of is such that I will see socket leaks? This is running against JRE 1.3.1 (on linux). I have a custy that is seeing leaks WebSphere and at the moment, httpclient is the current suspect since it's one of the few things in the app that causes it to open sockets. Thanks, -Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: 2.0 rc1 leaking sockets?
Hey Michael, Thanks .. I am using setMaxConnectionsPerHost()... and not setMaxTotalConnections(). Assuming there is a fixed number of hosts, will there be a fixed number of sockets used ? Also were there any API or behavioral changes (besides bug fixes :-)) between 2.0 rc1 and final that I should be aware of? Or is there a doc somewhere that has the list of changes between rc1 and final? Thanks again! -Eric Michael Becke wrote: On Mar 17, 2004, at 5:05 PM, Eric Bloch wrote: Thanks... just to be clear though... the multi-threadded connection mgr should never hold open more than a fixed number of sockets though, right? Yes, any one instance of MultiThreadedHttpConnectionManager will only have a certain number of open connections. This is configurable via setMaxTotalConnections(). You will need to ensure that you are reusing the same instance of the connection manager. There were also a number of bugs fixed in the MultiThreadedHttpConnectionManager between 2.0 RC1 and the final 2.0 relase. I would suggest upgrading to 2.0 just to be sure. Mike - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
2.0 rc1 leaking sockets?
Hi there, I've got a question. I'm using the Multithreaded Connection Manager and I know that I'm always calling releaseConnection() on every connection I use. Are there any reports (or possibilities) that releaseConnection() will fail to close an open socket or lose track of is such that I will see socket leaks? This is running against JRE 1.3.1 (on linux). I have a custy that is seeing leaks WebSphere and at the moment, httpclient is the current suspect since it's one of the few things in the app that causes it to open sockets. Thanks, -Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
what would an infinite loop in ChunkedInputStream.exhaustInputStreammean?
I am using the httpclient library to implement a caching, converting proxy and under some stress situations, I see this method infinite loop. Any clues? Thanks! -Eric - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
regression in cookie handling from 2.0 alpha3 to 2.0 rc 1 ?
I see that a lot of the cookie handling got cleaned up but there's one behavior that seems broken now. If I manually set a cookie header on a request (for example, if I'm proxying a request myself), HttpMethodBase will always clobber it during addCookieRequestHeader(). I would think that it should merge in any client state cookies to the header I add, rather than clobbering mine. Should I file a bug? FYI, I work-around this bug, by subclassing GetMethod (or any other HttpMethod) and overridding addCookieRequestHeader() to be a no-op. Thanks, Eric -- Eric Bloch Laszlo Systems, Inc. 1040 Mariposa Street, SF, CA 94107 voice: 415.241.2721 fax: 415.865.2914 email: [EMAIL PROTECTED] --- Laszlo allows Behr to deliver a breakthrough experience with ColorSmart by BEHR application. http://www.behr.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: regression in cookie handling from 2.0 alpha3 to 2.0 rc 1 ?
Hi Adrian, Thanks for the quick response! My problem is that I don't have a 'Cookie' object. I only have the text string for the name of the cookie and its value and I couldn't see any easy way for me to construct up a Cookie object from that... or any exposed http cookie header (not set-cookie header) parsing that would make it easy for me to construct up the Cookie object from my text (Im essentially reading a 'cookie' header myself and proxying the cookie over to another http server via the httpclient library). Parsing the cookie could actually be wasted cycles, too, because I don't maintain any state between requests; I create and destroy an HttpClient for each request (GetMethod) I execute. I'm happy with my current workaround, but it means I'll have to make sure the implementation bits don't change too much between revs, as you guys continue on. Again, thanks for your time and nice work! -Eric Adrian Sutton wrote: Hi Eric, If I manually set a cookie header on a request (for example, if I'm proxying a request myself), HttpMethodBase will always clobber it during addCookieRequestHeader(). I would think that it should merge in any client state cookies to the header I add, rather than clobbering mine. You should add cookies using the HttpState.addCookie method rather than adding it directly as a header, then it won't be clobbered and will be correctly merged into any other cookies being sent. We don't consider the current behaviour a bug, though if enough people requested it I imagine it would be possible to change. Regards, Adrian Sutton. -- Intencha tomorrow's technology today Ph: 38478913 0422236329 Suite 8/29 Oatland Crescent Holland Park West 4121 Australia QLD www.intencha.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Eric Bloch Laszlo Systems, Inc. 1040 Mariposa Street, SF, CA 94107 voice: 415.241.2721 fax: 415.865.2914 email: [EMAIL PROTECTED] --- Laszlo allows Behr to deliver a breakthrough experience with ColorSmart by BEHR application. http://www.behr.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
