Re: BindException problem in multithreaded http client
On Wed, 2008-09-03 at 16:03 -0400, Tomek Maciejewski wrote: > Hi Oleg, > > Thank for your reply. I have got the latest code from repository, but > the problem didn't disappear. I saved the logs here: > http://pentium.hopto.org/~thinred/files/output.zip > At the end of the crawler execution I started to get BindException. I > would appreciate if you could see this logs, because actually I don't > understand what is a relation between not reused connections and this > bind exception. What is the nature of this problem, and why it tries to > bind to 'address already in use'? Is there any way to solve this problem? > > Cheers, > Tomek > > It is all pretty simple. Please examine response messages in the log. Most of them (if not all) have "Connection: close" header, which means the server does not want the connection to be kept alive. As a result connections do not get reused. Every time a new request is executed, a new socket has to be open. Every time a new socket is open, a client port has to be allocated. Windows (per default) allocates the range from 1024 to 5000 for client ports. It usually takes a while before a used client port gets released back to the pool. If you execute 4000 requests in a tight loop you are pretty much bound to run out of client ports and end up with a bunch of BindExceptions Hope this helps Oleg - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Problem using HTTPClient with SSL in Websphere
On Wed, 2008-09-03 at 08:08 -0700, MarcEck wrote: > Hi, > > I already posted this problem in an ibm forum, but they haven't an answer > yet > > I have an implementation with HttpClient to make a https connection. > > When I use this with the JVM (in a junit test) from RSA (Rational Software > Architect), all works fine. If I call the same within websphere I get this > exception: > > javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted > certificate found > at com.ibm.jsse2.n.a(n.java:15) > at com.ibm.jsse2.jc.a(jc.java:171) > at com.ibm.jsse2.db.a(db.java:129) > at com.ibm.jsse2.db.a(db.java:333) > at com.ibm.jsse2.eb.a(eb.java:145) > at com.ibm.jsse2.eb.a(eb.java:274) > at com.ibm.jsse2.db.m(db.java:330) > at com.ibm.jsse2.db.a(db.java:149) > at com.ibm.jsse2.jc.a(jc.java:450) > at com.ibm.jsse2.jc.g(jc.java:115) > at com.ibm.jsse2.jc.a(jc.java:187) > at com.ibm.jsse2.j.write(j.java:8) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:88) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:127) > at java.io.FilterOutputStream.write(FilterOutputStream.java:97) > at > org.apache.commons.httpclient.methods.ByteArrayRequestEntity.writeRequest(ByteArrayRequestEntity.java:90) > > So I assume that the RSA JVM somehow can find the certificate, the websphere > not. But I checked the cacert files, they are both identical. > > Is there a way to tell HTTPClient which keystore it should use. So I can > create an own keystore, which I provide to HTTPClient? > > Thanks in advance ... > Please have a look at the "Customizing SSL in HttpClient" and "Examples of SSL customization in HttpClient" sections of the HttpClient SSL guide: http://hc.apache.org/httpclient-3.x/sslguide.html Oleg > /Marc > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: BindException problem in multithreaded http client
Hi Oleg,
Thank for your reply. I have got the latest code from repository, but
the problem didn't disappear. I saved the logs here:
http://pentium.hopto.org/~thinred/files/output.zip
At the end of the crawler execution I started to get BindException. I
would appreciate if you could see this logs, because actually I don't
understand what is a relation between not reused connections and this
bind exception. What is the nature of this problem, and why it tries to
bind to 'address already in use'? Is there any way to solve this problem?
Cheers,
Tomek
Oleg Kalnichevski wrote:
Tomek,
I improved some of the DEBUG log statements and reduced the amount of
noise in the logs. Feel free to get the latest code snapshot from SVN
trunk and retest with your application.
Cheers
Oleg
On Wed, 2008-09-03 at 12:27 +0200, Oleg Kalnichevski wrote:
On Tue, 2008-09-02 at 17:57 -0400, Tomek Maciejewski wrote:
Thanks Oleg, now it works. Earlier I used System.setProperty to enable
logging but I get only WARN level logging, I don't know if the
properties are case-sensitive because I set the same properties. Only
difference was that I wrote "debug" instead of "debug".
Generally, with the debug enabled I don't get the exception. Maybe it is
because, the crawler runs a little slower because of logging operation
and saving them to output?
Can be.
In the logs I found that the connection is created 3715 times and in the
client I have the limit of 100 connections. Generally I often see
something like that:
[DEBUG] ThreadSafeClientConnManager - Released connection open but not
marked reusable.
[DEBUG] DefaultClientConnection - Connection shut down
[DEBUG] ConnPoolByRoute - Freeing connection
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]][null]
[DEBUG] ConnPoolByRoute - Creating new connection
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]]
[DEBUG] ThreadSafeClientConnManager -
ThreadSafeClientConnManager.getConnection:
HttpRoute[{}->http://www.mrw.interscience.wiley.com], timeout = 0
So the connection is not marked as reusable? Could it be a reason of my
BindException problems?
Yes, this is precisely the reason. I cannot say why the connection got
marked as non-reusable in the first place without seeing the complete
context log.
Oleg
I execute connections very simultaneously, right now there may be about
600 threads in executors and every of them invokes 'execute method' on
httpclient object.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Problem using HTTPClient with SSL in Websphere
Hi, I already posted this problem in an ibm forum, but they haven't an answer yet I have an implementation with HttpClient to make a https connection. When I use this with the JVM (in a junit test) from RSA (Rational Software Architect), all works fine. If I call the same within websphere I get this exception: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: No trusted certificate found at com.ibm.jsse2.n.a(n.java:15) at com.ibm.jsse2.jc.a(jc.java:171) at com.ibm.jsse2.db.a(db.java:129) at com.ibm.jsse2.db.a(db.java:333) at com.ibm.jsse2.eb.a(eb.java:145) at com.ibm.jsse2.eb.a(eb.java:274) at com.ibm.jsse2.db.m(db.java:330) at com.ibm.jsse2.db.a(db.java:149) at com.ibm.jsse2.jc.a(jc.java:450) at com.ibm.jsse2.jc.g(jc.java:115) at com.ibm.jsse2.jc.a(jc.java:187) at com.ibm.jsse2.j.write(j.java:8) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:88) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:127) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) at org.apache.commons.httpclient.methods.ByteArrayRequestEntity.writeRequest(ByteArrayRequestEntity.java:90) So I assume that the RSA JVM somehow can find the certificate, the websphere not. But I checked the cacert files, they are both identical. Is there a way to tell HTTPClient which keystore it should use. So I can create an own keystore, which I provide to HTTPClient? Thanks in advance ... /Marc -- View this message in context: http://www.nabble.com/Problem-using-HTTPClient-with-SSL-in-Websphere-tp19291299p19291299.html Sent from the HttpClient-User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: BindException problem in multithreaded http client
Tomek,
I improved some of the DEBUG log statements and reduced the amount of
noise in the logs. Feel free to get the latest code snapshot from SVN
trunk and retest with your application.
Cheers
Oleg
On Wed, 2008-09-03 at 12:27 +0200, Oleg Kalnichevski wrote:
> On Tue, 2008-09-02 at 17:57 -0400, Tomek Maciejewski wrote:
> > Thanks Oleg, now it works. Earlier I used System.setProperty to enable
> > logging but I get only WARN level logging, I don't know if the
> > properties are case-sensitive because I set the same properties. Only
> > difference was that I wrote "debug" instead of "debug".
> > Generally, with the debug enabled I don't get the exception. Maybe it is
> > because, the crawler runs a little slower because of logging operation
> > and saving them to output?
>
> Can be.
>
> > In the logs I found that the connection is created 3715 times and in the
> > client I have the limit of 100 connections. Generally I often see
> > something like that:
> >
> > [DEBUG] ThreadSafeClientConnManager - Released connection open but not
> > marked reusable.
> > [DEBUG] DefaultClientConnection - Connection shut down
> > [DEBUG] ConnPoolByRoute - Freeing connection
> > [HttpRoute[{}->http://www.mrw.interscience.wiley.com]][null]
> > [DEBUG] ConnPoolByRoute - Creating new connection
> > [HttpRoute[{}->http://www.mrw.interscience.wiley.com]]
> > [DEBUG] ThreadSafeClientConnManager -
> > ThreadSafeClientConnManager.getConnection:
> > HttpRoute[{}->http://www.mrw.interscience.wiley.com], timeout = 0
> >
> > So the connection is not marked as reusable? Could it be a reason of my
> > BindException problems?
>
> Yes, this is precisely the reason. I cannot say why the connection got
> marked as non-reusable in the first place without seeing the complete
> context log.
>
> Oleg
>
> > I execute connections very simultaneously, right now there may be about
> > 600 threads in executors and every of them invokes 'execute method' on
> > httpclient object.
>
>
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: BindException problem in multithreaded http client
On Tue, 2008-09-02 at 17:57 -0400, Tomek Maciejewski wrote:
> Thanks Oleg, now it works. Earlier I used System.setProperty to enable
> logging but I get only WARN level logging, I don't know if the
> properties are case-sensitive because I set the same properties. Only
> difference was that I wrote "debug" instead of "debug".
> Generally, with the debug enabled I don't get the exception. Maybe it is
> because, the crawler runs a little slower because of logging operation
> and saving them to output?
Can be.
> In the logs I found that the connection is created 3715 times and in the
> client I have the limit of 100 connections. Generally I often see
> something like that:
>
> [DEBUG] ThreadSafeClientConnManager - Released connection open but not
> marked reusable.
> [DEBUG] DefaultClientConnection - Connection shut down
> [DEBUG] ConnPoolByRoute - Freeing connection
> [HttpRoute[{}->http://www.mrw.interscience.wiley.com]][null]
> [DEBUG] ConnPoolByRoute - Creating new connection
> [HttpRoute[{}->http://www.mrw.interscience.wiley.com]]
> [DEBUG] ThreadSafeClientConnManager -
> ThreadSafeClientConnManager.getConnection:
> HttpRoute[{}->http://www.mrw.interscience.wiley.com], timeout = 0
>
> So the connection is not marked as reusable? Could it be a reason of my
> BindException problems?
Yes, this is precisely the reason. I cannot say why the connection got
marked as non-reusable in the first place without seeing the complete
context log.
Oleg
> I execute connections very simultaneously, right now there may be about
> 600 threads in executors and every of them invokes 'execute method' on
> httpclient object.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
