Re: A bug?

[Yue Luo]

As you note, this is not really a problem as all IOExceptions caused 
during close are buried.  It would be nice if this worked a little more 
cleanly though.  I have no idea why we close all of the various streams 
and then the socket.  It does seems a little redundant.  Does closing 
the output not close the input in this case?
Closing the output may close the input too.  But this is not problem. 
You can usually close a closed stream.  The original problem is caused 
by flush().  Because the output stream is a buffered stream, when you 
close it, it will try to flush its content before really close.  This 
flush caused exception, not the close itself.

Yue

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: A bug?

[Michael Becke]

Hi Yue,

Yue Luo wrote:

Hi,
  Version 2.0-rc1
  When HttpConnection.closeSocketAndStream() is called, it first closes 
the input stream, and then it closes the output stream.  When I am using 
 JSSE, the output stream seems to be also closed by the underlying 
system when the input stream is closed.  So when this method tries to 
close the output stream ( a buffered stream), flush() method will be 
called first implicitly.  Then an IOException will be thrown because the 
underlying stream is considered closed.  WrappedOutputStream.flush() 
will catch the exception and then WrappedOutputStream.handleException() 
will be called to convert the exception to a HttpRecoverableException.
Yes, this is something we have noticed before.  For a while there we had 
a problem where closing a socket would cause an infinite loop because of 
this.  close->WrappedOutputStream.handleException->close->

  This exception is throw in perfect normal working condition, so I 
suppose it is a bug, although the end user will not notice it.  I would 
suggest closing output stream before closing input stream in 
HttpConnection.closeSocketAndStream().
  I am curious, is it really necessary to close the streams separately? 
 Or can we just close the socket directly?
As you note, this is not really a problem as all IOExceptions caused 
during close are buried.  It would be nice if this worked a little more 
cleanly though.  I have no idea why we close all of the various streams 
and then the socket.  It does seems a little redundant.  Does closing 
the output not close the input in this case?

Any thoughts on why we close the streams and then the socket?

  Another minor typo bug, the LOG message in 
HttpMethodBase.readResponseBody(HttpConnection conn) is wrong.  It is 
for the other method with the same name.  It is probably a copy & paste 
bug.
Quite right.

Thanks,

Mike

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

A bug?

[Yue Luo]

Hi,
  Version 2.0-rc1
  When HttpConnection.closeSocketAndStream() is called, it first closes 
the input stream, and then it closes the output stream.  When I am using 
 JSSE, the output stream seems to be also closed by the underlying 
system when the input stream is closed.  So when this method tries to 
close the output stream ( a buffered stream), flush() method will be 
called first implicitly.  Then an IOException will be thrown because the 
underlying stream is considered closed.  WrappedOutputStream.flush() 
will catch the exception and then WrappedOutputStream.handleException() 
will be called to convert the exception to a HttpRecoverableException.
  This exception is throw in perfect normal working condition, so I 
suppose it is a bug, although the end user will not notice it.  I would 
suggest closing output stream before closing input stream in 
HttpConnection.closeSocketAndStream().
  I am curious, is it really necessary to close the streams separately? 
 Or can we just close the socket directly?
  Another minor typo bug, the LOG message in 
HttpMethodBase.readResponseBody(HttpConnection conn) is wrong.  It is 
for the other method with the same name.  It is probably a copy & paste bug.
  Thanks.

Yue

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: strange behavior of 2.0-rc1

[Leo Galambos]

Hi Mike,

you have it here: http://com-os2.ms.mff.cuni.cz/temp/example.log.gz 
(2767808B)
I set maxTotalConnection=5.

-g-

Michael Becke wrote:

Hi Leo,

Attached at the bottom of my message is another log4j configuration.   
This should considerably decrease the amount of logging.  It also 
keeps  all of the logging info in a single file.  To successfully 
debug this  case we will need all of the logging data for a particular 
session.  In  conjunction with this debugging we also need to reduce 
the time it  takes for this error to come up.  As I suggested in my 
email a few day  ago you should also try decreasing the number of 
total connections via  
MultiThreadedHttpConnectionManager.setMaxTotalConnections().

Mike

log4j.rootLogger=WARN, R

log4j.appender.R=org.apache.log4j.FileAppender
log4j.appender.R.File=example.log
log4j.appender.R.append=false
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.logger.org.apache.commons.httpclient.HttpMethodBase=DEBUG
log4j.logger.org.apache.commons.httpclient.MultiThreadedHttpConnectionMa 
nager=DEBUG
log4j.logger.org.apache.commons.httpclient.HttpConnection=DEBUG

On Thursday, August 28, 2003, at 10:36 PM, Leo Galambos wrote:

Hi Mike,

I've never used log4j and I missed your logging.html page, thus I  
tried my simple config (see below). Unfortunately, I cannot 
reproduce  the bug easily (-> no test case yet) - the robot has to 
run more than  6 hours.

BTW: I had to rotate log files due to a huge volume of data, so you  
can see last 30 log files. I guess the format is obvious from the  
config file below. If you cannot find what you need, please, edit my  
log4j cfg file and I will rerun it again.

Thank you
Leo
--

My log4j config:
log4j.rootLogger=debug, R
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log
log4j.appender.R.MaxFileSize=1000KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=30
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.logger.org.apache.commons.httpclient=DEBUG


-
To unsubscribe, e-mail: 
[EMAIL PROTECTED]
For additional commands, e-mail: 
[EMAIL PROTECTED]






-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: strange behavior of 2.0-rc1

[Michael Becke]

Hi Leo,

Attached at the bottom of my message is another log4j configuration.   
This should considerably decrease the amount of logging.  It also keeps  
all of the logging info in a single file.  To successfully debug this  
case we will need all of the logging data for a particular session.  In  
conjunction with this debugging we also need to reduce the time it  
takes for this error to come up.  As I suggested in my email a few day  
ago you should also try decreasing the number of total connections via  
MultiThreadedHttpConnectionManager.setMaxTotalConnections().

Mike

log4j.rootLogger=WARN, R

log4j.appender.R=org.apache.log4j.FileAppender
log4j.appender.R.File=example.log
log4j.appender.R.append=false
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.logger.org.apache.commons.httpclient.HttpMethodBase=DEBUG
log4j.logger.org.apache.commons.httpclient.MultiThreadedHttpConnectionMa 
nager=DEBUG
log4j.logger.org.apache.commons.httpclient.HttpConnection=DEBUG

On Thursday, August 28, 2003, at 10:36 PM, Leo Galambos wrote:

Hi Mike,

I've never used log4j and I missed your logging.html page, thus I  
tried my simple config (see below). Unfortunately, I cannot reproduce  
the bug easily (-> no test case yet) - the robot has to run more than  
6 hours.

BTW: I had to rotate log files due to a huge volume of data, so you  
can see last 30 log files. I guess the format is obvious from the  
config file below. If you cannot find what you need, please, edit my  
log4j cfg file and I will rerun it again.

Thank you
Leo
--

My log4j config:
log4j.rootLogger=debug, R
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log
log4j.appender.R.MaxFileSize=1000KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=30
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.logger.org.apache.commons.httpclient=DEBUG


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: what would an infinite loop in ChunkedInputStream.exhaustInputStreammean?

[Ortwin Glück]

Adrian Sutton wrote:
My semi-educated guess would be that the server never stops sending data so
HttpClient continues to try to read it.  
Don't use HttpClient to get the MP3 stream from a web radio station :-)

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: what would an infinite loop inChunkedInputStream.exhaustInputStream mean?

[Adrian Sutton]

My semi-educated guess would be that the server never stops sending data so
HttpClient continues to try to read it.  This could either be because the
content-length was an extremely large number (larger than the size of an
int?) or perhaps because chunked encoding was used and the server never
sends the final closing chunk.

Probably an important question is: is it actually in an infinite loop or is
it waiting on data from the server?  It's possible that HttpClient expect
more content than the server sends and so they both wind up sitting their
waiting for each other until one of them eventually times out.

The answer to these questions and more probably lie in the wire trace log -
instructions on generating such a thing can be found at
http://jakarta.apache.org/commons/httpclient/logging.html

Hope that provides some hints anyway.

Regards,

Adrian Sutton.

On 29/08/2003 3:59 PM, "Eric Bloch" <[EMAIL PROTECTED]> wrote:

> I am using the httpclient library to implement a caching, converting
> proxy and under some stress situations, I see this method infinite loop.
> 
> Any clues?
> 
> Thanks!
> 
> -Eric
> 
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail:
> [EMAIL PROTECTED]
> 

--
Intencha "tomorrow's technology today"
Ph: 38478913 0422236329
Suite 8/29 Oatland Crescent
Holland Park West 4121
Australia QLD
www.intencha.com


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

what would an infinite loop in ChunkedInputStream.exhaustInputStreammean?

[Eric Bloch]

I am using the httpclient library to implement a caching, converting 
proxy and under some stress situations, I see this method infinite loop.

Any clues?

Thanks!

-Eric



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: strange behavior of 2.0-rc1

[Leo Galambos]

Hi Mike,

I've never used log4j and I missed your logging.html page, thus I tried 
my simple config (see below). Unfortunately, I cannot reproduce the bug 
easily (-> no test case yet) - the robot has to run more than 6 hours.

BTW: I had to rotate log files due to a huge volume of data, so you can 
see last 30 log files. I guess the format is obvious from the config 
file below. If you cannot find what you need, please, edit my log4j cfg 
file and I will rerun it again.

Thank you
Leo
--

My log4j config:
log4j.rootLogger=debug, R
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log
log4j.appender.R.MaxFileSize=1000KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=30
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
log4j.logger.org.apache.commons.httpclient=DEBUG

Michael Becke wrote:

Hi Leo,

Could you give me a little more detail about what these various files  
are?  Do each of them show the problem that's occurring?  Also, 
though  it's a little late now, we really only need the debug log for  
MultiThreadedHttpConnectionManager and perhaps the wire I think.  
Have  you had any luck in creating a test case?

Thanks,

Mike

On Thursday, August 28, 2003, at 01:45 PM, Leo Galambos wrote:

Mike, the the logs are here: http://com-os2.ms.mff.cuni.cz/temp/  
(example.*.*)

THX

-g-


Michael Becke wrote:

Yea, "good news" :)

So now we know that we're losing connections somewhere, and they  
are   not being garbage collected.  There are a bunch of things 
we  can try.Here are a few ideas:

 - enable debug logging for the 
MultiThreadedHttpConnectionManager. You will probably also 
want to print out the thread name with  the  log.   This will 
require Log4j or JRE 1.4 logging.
 - reduce the number of threads executing methods as well as the  
max   number of
connections(MultiThreadedHttpConnectionManager.setMaxTotalConnection 
s( )) .  This should cause the error to occur more quickly.
 - if possible, try to come up with something standalone that
reproduces the problem.  If I can reproduce the error I will be   
better  able to debug the problem.
 - run the application in debug mode so that when things get 
locked   up  we can see where each of the threads are.

Also, any more detail you can provide about your application 
andenvironment would be helpful.  Please let me know if you 
have any   ideas  or questions.

Mike

On Monday, August 25, 2003, at 01:59 PM, Leo Galambos wrote:

Hi Mike,

"good news":

c-0.log:org.apache.commons.httpclient.HttpException: Timeout  
waiting   for connection
c-0.log-at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.do 
Ge tC onnection(MultiThreadedHttpConnectionManager.java:314)
c-0.log-at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.ge 
tC on nection(MultiThreadedHttpConnectionManager.java:241)
c-0.log-at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.j 
av a: 609)
c-0.log-at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.j 
av a: 500)
c-0.log-at org.egothor.robot.Network.get(Network.java:109)
c-0.log-at   
org.egothor.robot.Capek$Fetcher.run(Capek.java:186)

What next?

THX

Leo

Michael Becke wrote:

Hi Leo,

These 20 CLOSE_WAIT sockets are from the 20 connections in the  
pool   held by MultiThreadedHttpConnectionManager.  When  
connections are   released they are not necessarily closed.

My guess is that connections are being lost somewhere.  If all 
20connections are checked out and not released the 21st 
request  for a   connection will block until one is available.  
You can  verify this  by  setting httpConnectionFactoryTimeout 
on  HttpClient.  This  timeout is  how long HttpClient will wait 
when  getting a connection  from the  connection manager.  
Setting this  timeout to something  big is  probably best, that 
way you won't  run into it by accident.

If connections are getting lost somewhere and
httpConnectionFactoryTimeout is set you will start getting
HttpExceptions with a message of "Timeout waiting for 
connection"   on  every method execute.  Please give this a shot 
to ensure  that  the  problem is with lost connections.  Once 
verified we  can start  looking  for a cause.

Also, in you example Network.java class I noticed the 
following   lines:

while (r.status == -1 && attempt++ < RETRIES) {
try {
r.status = client.executeMethod(method);
r.msg = null;
} catch (Throwable x) {
method.recycle();
r.msg = url + ": " + 

Re: strange behavior of 2.0-rc1

[Michael Becke]

Hi Leo,

Could you give me a little more detail about what these various files  
are?  Do each of them show the problem that's occurring?  Also, though  
it's a little late now, we really only need the debug log for  
MultiThreadedHttpConnectionManager and perhaps the wire I think.  Have  
you had any luck in creating a test case?

Thanks,

Mike

On Thursday, August 28, 2003, at 01:45 PM, Leo Galambos wrote:

Mike, the the logs are here: http://com-os2.ms.mff.cuni.cz/temp/  
(example.*.*)

THX

-g-


Michael Becke wrote:

Yea, "good news" :)

So now we know that we're losing connections somewhere, and they  
are   not being garbage collected.  There are a bunch of things we  
can try.Here are a few ideas:

 - enable debug logging for the MultiThreadedHttpConnectionManager.  
   You will probably also want to print out the thread name with  
the  log.   This will require Log4j or JRE 1.4 logging.
 - reduce the number of threads executing methods as well as the  
max   number of
connections(MultiThreadedHttpConnectionManager.setMaxTotalConnection 
s( )) .  This should cause the error to occur more quickly.
 - if possible, try to come up with something standalone that
reproduces the problem.  If I can reproduce the error I will be   
better  able to debug the problem.
 - run the application in debug mode so that when things get locked  
 up  we can see where each of the threads are.

Also, any more detail you can provide about your application and
environment would be helpful.  Please let me know if you have any   
ideas  or questions.

Mike

On Monday, August 25, 2003, at 01:59 PM, Leo Galambos wrote:

Hi Mike,

"good news":

c-0.log:org.apache.commons.httpclient.HttpException: Timeout  
waiting   for connection
c-0.log-at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.do 
Ge tC onnection(MultiThreadedHttpConnectionManager.java:314)
c-0.log-at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.ge 
tC on nection(MultiThreadedHttpConnectionManager.java:241)
c-0.log-at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.j 
av a: 609)
c-0.log-at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.j 
av a: 500)
c-0.log-at org.egothor.robot.Network.get(Network.java:109)
c-0.log-at   
org.egothor.robot.Capek$Fetcher.run(Capek.java:186)

What next?

THX

Leo

Michael Becke wrote:

Hi Leo,

These 20 CLOSE_WAIT sockets are from the 20 connections in the  
pool   held by MultiThreadedHttpConnectionManager.  When  
connections are   released they are not necessarily closed.

My guess is that connections are being lost somewhere.  If all 20  
  connections are checked out and not released the 21st request  
for a   connection will block until one is available.  You can  
verify this  by  setting httpConnectionFactoryTimeout on  
HttpClient.  This  timeout is  how long HttpClient will wait when  
getting a connection  from the  connection manager.  Setting this  
timeout to something  big is  probably best, that way you won't  
run into it by accident.

If connections are getting lost somewhere and
httpConnectionFactoryTimeout is set you will start getting
HttpExceptions with a message of "Timeout waiting for connection"  
 on  every method execute.  Please give this a shot to ensure  
that  the  problem is with lost connections.  Once verified we  
can start  looking  for a cause.

Also, in you example Network.java class I noticed the following   
lines:

while (r.status == -1 && attempt++ < RETRIES) {
try {
r.status = client.executeMethod(method);
r.msg = null;
} catch (Throwable x) {
method.recycle();
r.msg = url + ": " + x.getMessage();
if (logger.isLoggable(Level.FINE)) {
logger.log(Level.FINE, attempt + " of " +  
  url, x);
}
}
}

HttpMethod.recycle() resets all values in the method, including  
the   URL and request headers.  Calling recycle in the exception  
case  and  not resetting the various values will make the retry  
behave  quite  differently.  My guess is that this is not what  
you were  trying to >> do.

Mike

On Saturday, August 23, 2003, at 01:07 PM, Leo Galambos wrote:

Hi.

I am using httpclient (HC) in a webcrawler. After 6 hours of  
run,  HC  stops working and I think, it is locked by some lock  
of a  critical  section in HC. The "problematic" code, I use, is  
here:   http://www.egothor.org/temp/Network.java

When 20 threads are used, I have this in `netstat -an` after
stopping the robot from its console (note: the number of dead
connection is really *20*):
tcp23786  0 censored:60083censored2:80 
CLOSE_WAIT
tcp24179  0 censored:60080censored2:80 
CLOSE_WAIT
tcp24143  0 cens

DO NOT REPLY [Bug 22756] - npe when using a getmethod

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22756

npe when using a getmethod





--- Additional Comments From [EMAIL PROTECTED]  2003-08-29 02:03 ---
Hi Yashi,

The problem seems to be how you are trying to execute the GET.  Your sample code has 
some 
ordering problems, calling getResponseBodyAsStream() before execute().  It also does 
not use the 
HttpClient class to execute the method, which is highly discouraged.  Please take a 
look at the 
HttpClient tutorial[1] and the sample code[2] for how to execute a GetMethod.  Please 
let me know 
if you have any questions.

Mike

[1] http://jakarta.apache.org/commons/httpclient/tutorial.html
[2] http://cvs.apache.org/viewcvs.cgi/jakarta-commons/httpclient/src/examples/

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

DO NOT REPLY [Bug 22800] - Crashes when it gets a redirect

[bugzilla]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22800

Crashes when it gets a redirect





--- Additional Comments From [EMAIL PROTECTED]  2003-08-29 01:34 ---
Hi Adam,

Thank you for posting this bug.  I tried this out and was unfortunately unable to 
reproduce the 
problem.  Here's what I tried:

System.setProperty("org.apache.commons.logging.Log", 
"org.apache.commons.logging.impl.SimpleLog"); 
System.setProperty("org.apache.commons.logging.simplelog.showdatetime", 
"true"); 
System.setProperty("org.apache.commons.logging.simplelog.log.httpclient.wire", 
"debug"); 

System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.commons.httpclient",
 
"debug"); 

String url = "http://www.ibiblio.org/maven";;

HttpClient client = new HttpClient();
GetMethod get = new GetMethod(url);

client.executeMethod(get);
get.getResponseBody();

The request was redirected successfully.  There are also a number of test cases for 
redirects that 
work with HEAD.  Is must be something about how HttpClient is configured or being 
called.  Can 
you provide a test case and/or a wire log 
?

Thanks,

Mike

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]