RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





Henri, Dan,
There are some discussions below about the usage
of msg/rmsg. What did you guys finally decide
on that? Just reuse msg? I'm seeing the problem
Dan mentioned on message larger than the ajp13
buffer (8K).


thanks,
shinta


> -Original Message-
> From: GOMEZ Henri [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 13, 2001 4:49 PM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
> 
> 
> > 1) First off, I strongly believe that this work (which I think is an
> >excellent feature, BTW), belongs in the 3.3 branch, and 
> *not* in 3.2. 
> 
> +1, there was a message about including latest mod_jk 
> features/corrections
> back to Tomcat 3.2.x but TC 3.2 Release Manager and others clairly
> stated that TC 3.2.x must be just correction of bugs without any
> new features. I asked about back-porting mod_jk/ajp13 since now
> Apache 2.0 or upload works fine with mod_jk/ajp from TC 3.3 branches
> but are not in TC 3.2. The decision was not to provide corrections
> to TC 3.2.
>  
> >Although it feels like a bug that we need to restart Apache 
> >every time we restart TC, it's documented behavior, and I 
> >consider fixing it more of a feature enhancement.  
> >The scope of work you're doing here is considerable,
> >and the code you're modifying is complex (and involves lots of 
> >different possible situations).  I would not feel 
> comfortable committing 
> >this to 3.2 unless it had seen a *lot* of testing in the 3.3 
> branch.  (Am 
> >I incorrect in remembering that there has been discussion of 
> putting this
> into 3.2?)
> 
> Right many works, some testing but back-port could introduce 
> bugs in TC3.2.
> The decision was to not port mod_jk back to tc 3.2. But 
> people could still
> build mod_jk from Tomcat 3.3 and use against there Tomcat 
> 3.2.x. That's why
> I milite for a jakarta sub-project (web-connector). These 
> kind of connector
> work must outside Tomcat trees. (Votes ?)
> 
> > 2) Specific problems (line numbers by patch against the 3.3 
> code base)
> > 
> > - msg/rmsg: Here is how I think you want the code to work: 
> >msg always holds the basic request information, so, in case 
> of an error, we
> can 
> >resend the message.  rmsg is used as the buffer for response 
> information.
> 
> Dan you're our resident hacker/expert on mod_jk native part 
> 
> >This is a good idea, but there are a few problems -- first, and most
> >importantly, if there is POST data, the "msg" buffer 
> >immediately gets reused to send that information to TC (ll. 
> 596-600).  
> >In that case,  on line 691, if you retry after a connection 
> reset event, 
> >you'd be using a buffer filled with POST data instead of a 
> buffer with
> header information.  
> 
> I'm not too confortable with msg/rmsg that's why I asked for your code
> revue (and it's excellent ;-)
> 
> >Possibly you could fix this by calling 
> ajp13_marshal_into_msgb again when 
> >you retry -- I don't know how the state of the 
> web_server_service object
> (s), 
> >changes over time, so I don't know for sure if this would 
> work or not.  But
> 
> >that's where I'd look next.  
> 
> >Or you could pass rmsg into the sendrequest, and use that to
> >send the POST data over.  I dunno.
> 
> 
> >Your use of msg/rmsg in getreply seems faulty to me.  I don't 
> >think you need msg there at all, and I'm sure that you've 
> got a problem
> with 
> >mixing the two of them.  Specifically, on l. 636, you call 
> >
> >  rc = ajp13_process_callback(rmsg, p, s, l);
> >
> >But, then a few lines later, you do:
> >
> >  else if(JK_AJP13_HAS_RESPONSE == rc) {
> > rc = connection_tcp_send_message(p, msg, l);
> 
> Right, we must drop rmsg use and do all stuff with msg.
> 
> 
> >If you look at ajp13_process_callback, at the GET_BODY_CHUNK 
> >case, you'll see that it's reading request data into the 
> buffer passed in
> as a param
> >(which is what getreply is calling 'rmsg').  But then, when you call
> >connection_tcp_send_message, you'll be sending over whatever 
> >was in msg, rather than what the web server has read from 
> the browser.  
> 
> >You can fix this by changing connection_tcp_send_message to 
> use 'rmsg' (and
> 
> >then you don't need to pass msg into getreply at all).
> 
> >This problem will happen in case of a file upload, or 

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

> 1) First off, I strongly believe that this work (which I think is an
>excellent feature, BTW), belongs in the 3.3 branch, and *not* in 3.2. 

+1, there was a message about including latest mod_jk features/corrections
back to Tomcat 3.2.x but TC 3.2 Release Manager and others clairly
stated that TC 3.2.x must be just correction of bugs without any
new features. I asked about back-porting mod_jk/ajp13 since now
Apache 2.0 or upload works fine with mod_jk/ajp from TC 3.3 branches
but are not in TC 3.2. The decision was not to provide corrections
to TC 3.2.
 
>Although it feels like a bug that we need to restart Apache 
>every time we restart TC, it's documented behavior, and I 
>consider fixing it more of a feature enhancement.  
>The scope of work you're doing here is considerable,
>and the code you're modifying is complex (and involves lots of 
>different possible situations).  I would not feel comfortable committing 
>this to 3.2 unless it had seen a *lot* of testing in the 3.3 branch.  (Am 
>I incorrect in remembering that there has been discussion of putting this
into 3.2?)

Right many works, some testing but back-port could introduce bugs in TC3.2.
The decision was to not port mod_jk back to tc 3.2. But people could still
build mod_jk from Tomcat 3.3 and use against there Tomcat 3.2.x. That's why
I milite for a jakarta sub-project (web-connector). These kind of connector
work must outside Tomcat trees. (Votes ?)

> 2) Specific problems (line numbers by patch against the 3.3 code base)
> 
> - msg/rmsg: Here is how I think you want the code to work: 
>msg always holds the basic request information, so, in case of an error, we
can 
>resend the message.  rmsg is used as the buffer for response information.

Dan you're our resident hacker/expert on mod_jk native part 

>This is a good idea, but there are a few problems -- first, and most
>importantly, if there is POST data, the "msg" buffer 
>immediately gets reused to send that information to TC (ll. 596-600).  
>In that case,  on line 691, if you retry after a connection reset event, 
>you'd be using a buffer filled with POST data instead of a buffer with
header information.  

I'm not too confortable with msg/rmsg that's why I asked for your code
revue (and it's excellent ;-)

>Possibly you could fix this by calling ajp13_marshal_into_msgb again when 
>you retry -- I don't know how the state of the web_server_service object
(s), 
>changes over time, so I don't know for sure if this would work or not.  But

>that's where I'd look next.  

>Or you could pass rmsg into the sendrequest, and use that to
>send the POST data over.  I dunno.


>Your use of msg/rmsg in getreply seems faulty to me.  I don't 
>think you need msg there at all, and I'm sure that you've got a problem
with 
>mixing the two of them.  Specifically, on l. 636, you call 
>
>  rc = ajp13_process_callback(rmsg, p, s, l);
>
>But, then a few lines later, you do:
>
>  else if(JK_AJP13_HAS_RESPONSE == rc) {
>   rc = connection_tcp_send_message(p, msg, l);

Right, we must drop rmsg use and do all stuff with msg.


>If you look at ajp13_process_callback, at the GET_BODY_CHUNK 
>case, you'll see that it's reading request data into the buffer passed in
as a param
>(which is what getreply is calling 'rmsg').  But then, when you call
>connection_tcp_send_message, you'll be sending over whatever 
>was in msg, rather than what the web server has read from the browser.  

>You can fix this by changing connection_tcp_send_message to use 'rmsg' (and

>then you don't need to pass msg into getreply at all).

>This problem will happen in case of a file upload, or whenever there is
>enough POST data to exceed a single ajp13 buffer (8K).  

>Furthermore, in that
>case, I'm not sure if there's going to be any way to restart 
>the connection
>intelligently.  If the browser has already sent over 500 K of a 1M file
>upload when TC dies, I think we need to just kill that 
>request, rather than
>trying to restart it. I don't think the request is 
>recoverable. I'm not sure
>how to detect this in your code, but I think it needs to be 
>thought about.

Upload are a real problem since we just couldn't replay the upload
in case of Tomcat failure. Solution could be to ask browser to
resend but HOW (HTTP expert needed here)

> - The for loop on l. 689 makes me nervous.  Could you rewrite that:
>int i;
>for(i = 0 ; i < JK_RETRIES ; i++)
>That way you won't get an infinite loop if JK_RETRIES < 0.  
>And I think it's
>more idiomatic.  But, hey, that's up to you ;-).

+1

> - Do you understand the JK_METHOD macro?  I don't fully, but 
>take a look in
>jk_global.h.  In Win32 and Netware environments, this macro is set to
>"__stdcall" -- is this only needed for things which are called 
>from outside
>this file (like service())?  If neither of us understand it, 

JK_METHOD (stdcall) is needed when used you use functions
from outside but in our case it's static and so it's safe
to not set the JK_METHOD for sendrequest/getrep

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown



I proposed to [EMAIL PROTECTED] list some days 
now
that we start a subproject called web-connector where 

we could put mod_jk/mod_jserv/mod_webapp with 
both
native (Apache, NES, IIS) and Java 
(Modules/Interceptors for 
TC 3.2.x, 3.3 and 4.0)
 
Still no response but I really think we must 
externalize
that connector stuff from Tomcat's core code 
;-)
 
 
 >  1) 
First off, I strongly believe that this work (which I think is an 
> excellent feature, BTW), belongs in the 3.3 branch, and 
*not* in 3.2. > Although it feels like a bug that we 
need to restart Apache > every time we restart TC, 
it's documented behavior, and I > consider fixing it 
more of a feature enhancement.  The scope > of 
work you're doing here is considerable, and the code you're > modifying is complex (and involves lots of different possible 
> situations).  I would not feel comfortable 
committing this to 3.2 > unless it had seen a *lot* 
of testing in the 3.3 branch.  (Am > I incorrect 
in remembering that there has been discussion of > 
putting this into 3.2?) 

  Dan, sorry for not clarifying. I'm back porting the changes to 
  my own company's CVS, not the Apache Tomcat's CVS. We 
  have decided that we need this changes on our 3.2.1 
  Tomcat release but can't afford to release Tomcat 3.3 
  beta right now. 
  So you can definitely check in the changes to 3.3 branch, 
  and not 3.2.1. I think we've discussed that and that 
  was agreed. 
  thanks, shinta   

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





>  1) First off, I strongly believe that this work (which I think is an
> excellent feature, BTW), belongs in the 3.3 branch, and *not* in 3.2. 
> Although it feels like a bug that we need to restart Apache 
> every time we restart TC, it's documented behavior, and I 
> consider fixing it more of a feature enhancement.  The scope 
> of work you're doing here is considerable, and the code you're 
> modifying is complex (and involves lots of different possible 
> situations).  I would not feel comfortable committing this to 3.2
> unless it had seen a *lot* of testing in the 3.3 branch.  (Am 
> I incorrect in remembering that there has been discussion of 
> putting this into 3.2?)


Dan, sorry for not clarifying. I'm back porting the changes to my 
own company's CVS, not the Apache Tomcat's CVS. We have decided
that we need this changes on our 3.2.1 Tomcat release but can't 
afford to release Tomcat 3.3 beta right now.


So you can definitely check in the changes to 3.3 branch, and
not 3.2.1. I think we've discussed that and that was agreed.


thanks,
shinta
 

Re: FW: problem w/ ajp13 - if Tomcat is shutdown

[Dan Milstein]

Henri,

I looked in the archives and found the patch which I hadn't seen (my ISP's
mail server needs a serious talking to!).

Okay, I think I see some fairly serious issues.  I'll lay out here what I've
seen so far:

 1) First off, I strongly believe that this work (which I think is an
excellent feature, BTW), belongs in the 3.3 branch, and *not* in 3.2. 
Although it feels like a bug that we need to restart Apache every time we
restart TC, it's documented behavior, and I consider fixing it more of a
feature enhancement.  The scope of work you're doing here is considerable,
and the code you're modifying is complex (and involves lots of different
possible situations).  I would not feel comfortable committing this to 3.2
unless it had seen a *lot* of testing in the 3.3 branch.  (Am I incorrect in
remembering that there has been discussion of putting this into 3.2?)


 2) Specific problems (line numbers by patch against the 3.3 code base)
 
 - msg/rmsg: Here is how I think you want the code to work: msg always holds
the basic request information, so, in case of an error, we can resend the
message.  rmsg is used as the buffer for response information.

This is a good idea, but there are a few problems -- first, and most
importantly, if there is POST data, the "msg" buffer immediately gets reused
to send that information to TC (ll. 596-600).  In that case, on line 691, if
you retry after a connection reset event, you'd be using a buffer filled
with POST data instead of a buffer with header information.  Possibly you
could fix this by calling ajp13_marshal_into_msgb again when you retry -- I
don't know how the state of the web_server_service object (s), changes over
time, so I don't know for sure if this would work or not.  But that's where
I'd look next.  Or you could pass rmsg into the sendrequest, and use that to
send the POST data over.  I dunno.

Your use of msg/rmsg in getreply seems faulty to me.  I don't think you need
msg there at all, and I'm sure that you've got a problem with mixing the two
of them.  Specifically, on l. 636, you call 

  rc = ajp13_process_callback(rmsg, p, s, l);

But, then a few lines later, you do:

  else if(JK_AJP13_HAS_RESPONSE == rc) {
rc = connection_tcp_send_message(p, msg, l);

If you look at ajp13_process_callback, at the GET_BODY_CHUNK case, you'll
see that it's reading request data into the buffer passed in as a param
(which is what getreply is calling 'rmsg').  But then, when you call
connection_tcp_send_message, you'll be sending over whatever was in msg,
rather than what the web server has read from the browser.  You can fix this
by changing connection_tcp_send_message to use 'rmsg' (and then you don't
need to pass msg into getreply at all).

This problem will happen in case of a file upload, or whenever there is
enough POST data to exceed a single ajp13 buffer (8K).  Furthermore, in that
case, I'm not sure if there's going to be any way to restart the connection
intelligently.  If the browser has already sent over 500 K of a 1M file
upload when TC dies, I think we need to just kill that request, rather than
trying to restart it. I don't think the request is recoverable. I'm not sure
how to detect this in your code, but I think it needs to be thought about.

 - The for loop on l. 689 makes me nervous.  Could you rewrite that:
int i;
for(i = 0 ; i < JK_RETRIES ; i++)

That way you won't get an infinite loop if JK_RETRIES < 0.  And I think it's
more idiomatic.  But, hey, that's up to you ;-).

 - Do you understand the JK_METHOD macro?  I don't fully, but take a look in
jk_global.h.  In Win32 and Netware environments, this macro is set to
"__stdcall" -- is this only needed for things which are called from outside
this file (like service())?  If neither of us understand it, I'd like to
have people on Win32 and Netware systems test this (maybe they could even
explain why that call is needed -- and then we could document it!)  Maybe
you should be using it with sendrequest and getreply?

 - A few logging inconsistencies -- you call jk_log(l, JK_LOG_DEBUG,...),
when I think you want JK_LOG_ERROR.  Check l. 642, 702. 

 - Error/log message inconsistencies: 
   - l. 632 You mean "error reading response", I think
   - l. 642 You have "1" where you want ":", I think.

 - You seem to be using 4-character tabs -- the rest of the C-code (AFAIK)
uses 8-character tabs.  I think that's our standard (it is on the Java side,
for sure).  In any event, this is making your new code render bizarrely in
my editor.

 - Style notes (I'm style obsessive, I admit).  I would make it
ajp13_send_request (or at least send_request) and ajp13_get_reply (or
get_reply).

-Dan

GOMEZ Henri wrote:
> 
> >In the mean time, I have taken Henri's changes and back
> >port it to 3.2.1 (because I need it on 3.2.1). Everything
> >seems to work well. I've tested it in the normal scenarios
> >(one Apache, one Tomcat) and in the load-balanced scenarios.
> 
> Thanks for using the patch and tested it under LoadB

Re: FW: problem w/ ajp13 - if Tomcat is shutdown

[Dan Milstein]

Henri,

I think you're right about not trying to clean out the cache of endpoints. 
Too much work and not really necessary.

I'd be happy to check your code, but I don't think I've seen it -- did you
submit it to the list (I may have missed a few messages to the list)?

-Dan

GOMEZ Henri wrote:
> 
> >In the mean time, I have taken Henri's changes and back
> >port it to 3.2.1 (because I need it on 3.2.1). Everything
> >seems to work well. I've tested it in the normal scenarios
> >(one Apache, one Tomcat) and in the load-balanced scenarios.
> 
> Thanks for using the patch and tested it under LoadBalanced
> Tomcats.
> 
> >In the load-balanced scenarios, when I restart TC worker 1,
> >the code properly close the dead sockets and re-establish
> >new ones to the same worker (TC worker 1). The good
> >connections to TC worker 2 are untouched. They stay
> >connected.
> 
> Normal procedure
> 
> >I did notice something wierd. But this is un-related
> >to the code edits. This happens with or without Henri's
> >changes. When I restart TC worker 1, but shut down TC
> >worker 2, requests that supposed to go to TC worker 2
> >(because they belong to the same session, thus the load
> >balancer try to foward it to the same TC worker 2) took
> >sometime to get forwarded to TC worker 1. This maybe
> >another one of those "improvements" that can be done
> >to the load balancer worker.
> 
>  No problem here, when you shutdown a Tomcat in a
> load balancing architecture, you got request goes
> to the second even if there is a JVMROUTE set .
> 
> >Anyway, I'm pretty happy with Henri's changes. (Thanks
> >Henri!). Henri, are you going to check in the changes?
> 
> I'd like to see Dan check it since I created a second memory
> pool rmsg but I'm not too confident on it :
> 
> +rmsg = jk_b_new(&(p->pool));
> +jk_b_set_buffer_size( rmsg, DEF_BUFFER_SZ);
> + jk_b_reset(rmsg);
> 
> See you
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, email: [EMAIL PROTECTED]

-- 

Dan Milstein // [EMAIL PROTECTED]

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

>In the mean time, I have taken Henri's changes and back 
>port it to 3.2.1 (because I need it on 3.2.1). Everything 
>seems to work well. I've tested it in the normal scenarios 
>(one Apache, one Tomcat) and in the load-balanced scenarios.  

Thanks for using the patch and tested it under LoadBalanced 
Tomcats. 

>In the load-balanced scenarios, when I restart TC worker 1, 
>the code properly close the dead sockets and re-establish 
>new ones to the same worker (TC worker 1). The good 
>connections to TC worker 2 are untouched. They stay 
>connected.  

Normal procedure 

>I did notice something wierd. But this is un-related 
>to the code edits. This happens with or without Henri's 
>changes. When I restart TC worker 1, but shut down TC 
>worker 2, requests that supposed to go to TC worker 2 
>(because they belong to the same session, thus the load 
>balancer try to foward it to the same TC worker 2) took 
>sometime to get forwarded to TC worker 1. This maybe 
>another one of those "improvements" that can be done 
>to the load balancer worker.  

 No problem here, when you shutdown a Tomcat in a
load balancing architecture, you got request goes
to the second even if there is a JVMROUTE set .

>Anyway, I'm pretty happy with Henri's changes. (Thanks 
>Henri!). Henri, are you going to check in the changes?  

I'd like to see Dan check it since I created a second memory
pool rmsg but I'm not too confident on it :

+rmsg = jk_b_new(&(p->pool));
+jk_b_set_buffer_size( rmsg, DEF_BUFFER_SZ); 
+ jk_b_reset(rmsg);

See you

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





Hi, Dan,
I may not understand all of the issues here. But I really
don't think we should close all connections when we
detect one ECONNRESET. In my opinion, it is not necessary
to close all connections, since the next ECONNRESET 
will close the proper dead socket, anyway. It's not
needed to add all of those complexity.


In the mean time, I have taken Henri's changes and back
port it to 3.2.1 (because I need it on 3.2.1). Everything
seems to work well. I've tested it in the normal scenarios
(one Apache, one Tomcat) and in the load-balanced scenarios.


In the load-balanced scenarios, when I restart TC worker 1, 
the code properly close the dead sockets and re-establish
new ones to the same worker (TC worker 1). The good 
connections to TC worker 2 are untouched. They stay 
connected.


I did notice something wierd. But this is un-related
to the code edits. This happens with or without Henri's
changes. When I restart TC worker 1, but shut down TC 
worker 2, requests that supposed to go to TC worker 2 
(because they belong to the same session, thus the load
balancer try to foward it to the same TC worker 2) took 
sometime to get forwarded to TC worker 1. This maybe
another one of those "improvements" that can be done
to the load balancer worker.


Anyway, I'm pretty happy with Henri's changes. (Thanks
Henri!). Henri, are you going to check in the changes?


Let me know if I can do something else to help for
this case.


shinta


> -Original Message-
> From: Dan Milstein [mailto:[EMAIL PROTECTED]]
> Sent: Friday, March 09, 2001 3:08 PM
> To: [EMAIL PROTECTED]
> Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown
> 
> 
> In terms of invalidating the cache:
> 
> The jk_ajp13_worker objects hold onto a cache of endpoints -- 
> ep_cache.  It
> would be relatively simple to loop through this cache and 
> close all the
> connections in case of ECONNRESET (you do have to call a 
> macro to enter a
> critical section -- take a look at reuse_connection()).
> 
> However, this cache only holds onto endpoints which are *not* 
> being used. 
> When an endpoint is checked out of the cache (by 
> get_endpoint), or if the
> open socket descriptor is transfered to another endpoint (in
> reuse_connection), that connection is replaced by NULL in the cache.
> 
> So if we shut down all the connections in the cache, we won't 
> shut down the
> other connections which are handling requests at that moment. 
>  My only fear
> then is that, when those connections get their own ECONNRESET 
> errors, they,
> too, will try to shutdown all the connections in the cache.  
> If TC hasn't
> come back up yet, this won't be a problem, because there won't be any
> connections in the cache.  But it does make me a bit nervous.
> 
> Hope that's helpful...
> 
> -Dan
> 
> 
> 
> GOMEZ Henri wrote:
> > 
> > La prise de conscience de votre propre ignorance est un 
> grand pas vers la
> > connaissance.
> > -- Benjamin Disraeli
> > 
> > 
> > >-Original Message-
> > >From: Dan Milstein [mailto:[EMAIL PROTECTED]]
> > >Sent: Friday, March 09, 2001 6:34 AM
> > >To: [EMAIL PROTECTED]
> > >Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown
> > >
> > >
> > >Henri,
> > >
> > >You say that checking errno isn't safe in a multithreaded env
> > >(which would
> > >certainly makes sense to me, since it looks like a global var).
> > >
> > >However, after searching online, and reading up in
> > >"Programming Threads", by
> > >Kleiman, Shah and Smaalders, I find on p. 47:
> > >
> > >"Each thread has its own independent version of the errno
> > >variable.  This
> > >allows different threads to make system calls that may change
> > >the value of errno without interfering with each other."
> > >They are describing Posix threads.  "errno" is actually a
> > >macro, apparently, which accesses the correct, 
> thread-specific errno
> > variable.
> > 
> > Right, I checked in Linux errno.h for pthread
> > 
> > >Now, I am the first to admit that I don't understand all the weird
> > >intersections between threads and sockets in C, but this looks
> > >to me like checking errno against ECONNRESET may be fine.
> > 
> > More generally when you got a read error on TCP/IP stream
> > you could consider that the link to your server (tomcat) is broken :
> > 
> > - no more route to tomcat (broken lan or routers)
> > - server not working

Re: FW: problem w/ ajp13 - if Tomcat is shutdown

[Dan Milstein]

In terms of invalidating the cache:

The jk_ajp13_worker objects hold onto a cache of endpoints -- ep_cache.  It
would be relatively simple to loop through this cache and close all the
connections in case of ECONNRESET (you do have to call a macro to enter a
critical section -- take a look at reuse_connection()).

However, this cache only holds onto endpoints which are *not* being used. 
When an endpoint is checked out of the cache (by get_endpoint), or if the
open socket descriptor is transfered to another endpoint (in
reuse_connection), that connection is replaced by NULL in the cache.

So if we shut down all the connections in the cache, we won't shut down the
other connections which are handling requests at that moment.  My only fear
then is that, when those connections get their own ECONNRESET errors, they,
too, will try to shutdown all the connections in the cache.  If TC hasn't
come back up yet, this won't be a problem, because there won't be any
connections in the cache.  But it does make me a bit nervous.

Hope that's helpful...

-Dan



GOMEZ Henri wrote:
> 
> La prise de conscience de votre propre ignorance est un grand pas vers la
> connaissance.
> -- Benjamin Disraeli
> 
> 
> >-Original Message-
> >From: Dan Milstein [mailto:[EMAIL PROTECTED]]
> >Sent: Friday, March 09, 2001 6:34 AM
> >To: [EMAIL PROTECTED]
> >Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown
> >
> >
> >Henri,
> >
> >You say that checking errno isn't safe in a multithreaded env
> >(which would
> >certainly makes sense to me, since it looks like a global var).
> >
> >However, after searching online, and reading up in
> >"Programming Threads", by
> >Kleiman, Shah and Smaalders, I find on p. 47:
> >
> >"Each thread has its own independent version of the errno
> >variable.  This
> >allows different threads to make system calls that may change
> >the value of errno without interfering with each other."
> >They are describing Posix threads.  "errno" is actually a
> >macro, apparently, which accesses the correct, thread-specific errno
> variable.
> 
> Right, I checked in Linux errno.h for pthread
> 
> >Now, I am the first to admit that I don't understand all the weird
> >intersections between threads and sockets in C, but this looks
> >to me like checking errno against ECONNRESET may be fine.
> 
> More generally when you got a read error on TCP/IP stream
> you could consider that the link to your server (tomcat) is broken :
> 
> - no more route to tomcat (broken lan or routers)
> - server not working (tomcat was stopped or server restarted)
> 
> >Are there platforms where that's not true?
> 
> I've no idea but we migth have problems in differents interpretation
> of platform.
> 
> >The nice thing about getting that ECONNRESET error, is it lets
> >us go ahead and close out that connection, and try another one.
> 
> Done.
> 
> >We could even close out a whole cache of connections,
> >which would most likely be the right thing to do.
> 
> Good idea, I'll find how to do that.
> 
> >If we loop/retry, than how do we know to close the connection?
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, email: [EMAIL PROTECTED]

-- 

Dan Milstein // [EMAIL PROTECTED]

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

La prise de conscience de votre propre ignorance est un grand pas vers la
connaissance.
-- Benjamin Disraeli
 

>-Original Message-
>From: Dan Milstein [mailto:[EMAIL PROTECTED]]
>Sent: Friday, March 09, 2001 6:34 AM
>To: [EMAIL PROTECTED]
>Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown
>
>
>Henri,
>
>You say that checking errno isn't safe in a multithreaded env 
>(which would
>certainly makes sense to me, since it looks like a global var).
>
>However, after searching online, and reading up in 
>"Programming Threads", by
>Kleiman, Shah and Smaalders, I find on p. 47:
>
>"Each thread has its own independent version of the errno 
>variable.  This
>allows different threads to make system calls that may change 
>the value of errno without interfering with each other."
>They are describing Posix threads.  "errno" is actually a 
>macro, apparently, which accesses the correct, thread-specific errno
variable.

Right, I checked in Linux errno.h for pthread

>Now, I am the first to admit that I don't understand all the weird
>intersections between threads and sockets in C, but this looks 
>to me like checking errno against ECONNRESET may be fine.

More generally when you got a read error on TCP/IP stream
you could consider that the link to your server (tomcat) is broken :

- no more route to tomcat (broken lan or routers)
- server not working (tomcat was stopped or server restarted)

>Are there platforms where that's not true? 

I've no idea but we migth have problems in differents interpretation
of platform.

>The nice thing about getting that ECONNRESET error, is it lets 
>us go ahead and close out that connection, and try another one.  

Done.

>We could even close out a whole cache of connections, 
>which would most likely be the right thing to do.  

Good idea, I'll find how to do that.

>If we loop/retry, than how do we know to close the connection?



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

Here how we could do :


>Okay, I basically agree with you. I'll take out 
>the check for errno and just have recv() == -1 
>be considered a recoverable error (i.e: retry it). 
>However, I disagree with making the retry in a 
>loop for RETRIES times. This is because if one 
>retry fails, this means this error condition may 
>not be recoverable without any human interventions. 

We must retry against eventual other workers in a
LoadBalancing system. I must verify that my patch
allow that.

>What is the point of retrying more than once? 

No problem, I'll retry 3 times. It's a special
case, exception, so we could spend some time to
re-establish the connection to a working unit.

>My goal is not to wait for TC to come back up 
>or to wait for TC to be in a good state. My goal 
>is, if TC is in a good state already, why tell 
>the caller that it's an error. 

+1

>Opinions? 

I agree, with the new version you could see that we try
X time to send the request. It wasn't the case previously
which make me think that load-balancing was only working
on active server, but failed completly when one of then
was stopped.

We 


 jk_ajp13_worker.c.diff

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

Re: FW: problem w/ ajp13 - if Tomcat is shutdown

[Dan Milstein]

Henri,

You say that checking errno isn't safe in a multithreaded env (which would
certainly makes sense to me, since it looks like a global var).

However, after searching online, and reading up in "Programming Threads", by
Kleiman, Shah and Smaalders, I find on p. 47:

"Each thread has its own independent version of the errno variable.  This
allows different threads to make system calls that may change the value of
errno without interfering with each other."

They are describing Posix threads.  "errno" is actually a macro, apparently,
which accesses the correct, thread-specific errno variable.

Now, I am the first to admit that I don't understand all the weird
intersections between threads and sockets in C, but this looks to me like
checking errno against ECONNRESET may be fine.

Are there platforms where that's not true? 

The nice thing about getting that ECONNRESET error, is it lets us go ahead
and close out that connection, and try another one.  We could even close out
a whole cache of connections, which would most likely be the right thing to
do.  If we loop/retry, than how do we know to close the connection?

-Dan

GOMEZ Henri wrote:
> 
> >Okay so this means, you would prefer my proposed solution #1?
> >That was my inclination too.
> 
> Proposed solution #1 without the errno check.
> 
> My idea :
> 
> get the service code in a loop
> 
> for (i = 0; i < RETRIES; i++) {
> 
> if (send_request() < 0)
> continue;
> 
> if (read_reply() == 0)
> break;
> }
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, email: [EMAIL PROTECTED]

-- 

Dan Milstein // [EMAIL PROTECTED]

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





Okay, I basically agree with you. I'll take out 
the check for errno and just have recv() == -1 
be considered a recoverable error (i.e: retry it).


However, I disagree with making the retry in a
loop for RETRIES times. This is because if one 
retry fails, this means this error condition may 
not be recoverable without any human interventions.
What is the point of retrying more than once?


My goal is not to wait for TC to come back up
or to wait for TC to be in a good state. My goal 
is, if TC is in a good state already, why tell 
the caller that it's an error. 


Opinions?
shinta


> -Original Message-
> From: GOMEZ Henri [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, March 08, 2001 5:12 PM
> To: [EMAIL PROTECTED]
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
> 
> 
> >Okay so this means, you would prefer my proposed solution #1? 
> >That was my inclination too.  
> 
> Proposed solution #1 without the errno check.
> 
> My idea :
> 
> get the service code in a loop 
> 
> for (i = 0; i < RETRIES; i++) {
> 
>   if (send_request() < 0)
>       continue;
> 
>   if (read_reply() == 0)
>       break;
> }
>   
> 
> 
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, email: [EMAIL PROTECTED]
> 

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

>Okay so this means, you would prefer my proposed solution #1? 
>That was my inclination too.  

Proposed solution #1 without the errno check.

My idea :

get the service code in a loop 

for (i = 0; i < RETRIES; i++) {

if (send_request() < 0)
continue;

if (read_reply() == 0)
break;
}



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





> I revue your patch and I notice : 
>
> We could use select to determine if something happen to the 
> connection, typically readfds will be set if something is to read. 
> With ajp13 protocol where everything is consumed after reading the 
> previous tomcat reply, select will set read when the connection is 
> closed (in TCP a closed tcp session is really a message send from 
> closer). 
>
> But : 
> - Win32 platform are not handled with the select; 


This is because I haven't been able to reproduce this
on Windows 2000. I admit I didn't try very hard last
time. So I will try it again.


> - there is a check of errno which couldn't be safe in 
> multi-threaded env like Apache 2.0 


Any suggestions on how to do it better? The reason
I check for ECONNRESET is I want to do this only
if the error was caused by previously shut TC.
Any other recv() error should return Internal
Server Error, because they may not be recoverable.


> - if you remove your ethernet cable you may never see 
> anything in select readdfs before TCP/IP stack timeout.


well, I wasn't trying to handle that condition, though.
The select() should still be harmless.


> - Why loose cpu cycle to check if the communication socket is 
> available before each request to be sent. I'll be to use an 
> exception mecanism, so 
> - send request 
> - read reply 
> - if reply read fail, retry to send request, 
> if it fail = ERR 500 


Okay so this means, you would prefer my proposed solution #1?
That was my inclination too.


> I'll send some modification to worker later 

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[GOMEZ Henri]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





I revue your patch and I notice :


We could use select to determine if something happen to the
connection, typically readfds will be set if something is to read.
With ajp13 protocol where everything is consumed after reading the 
previous tomcat reply, select will set read when the connection is
closed (in TCP a closed tcp session is really a message send from
closer).


But :


- Win32 platform are not handled with the select;
- there is a check of errno which couldn't be safe in multi-threaded env like Apache 2.0
- if you remove your ethernet cable you may never see anything in select readdfs
  before TCP/IP stack timeout.


- Why loose cpu cycle to check if the communication socket is available before
  each request to be sent. I'll be to use an exception mecanism, so 


    - send request 
    - read reply


        - if reply read fail, retry to send request, if it fail = ERR 500


I'll send some modification to worker later

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





> BTW we should redirect this to the tomcat-users list as I don't think
> its a development issue.


Sorry about that. 


> I don't see anything wrong, and the only drastic difference from my 
> previous setup was that I had used the default 8007 and 8009.  Now 
> This is purely a guess, but I seem to remember reading that ports 
> >1 were special somehow and required root access for some 
> functions. What I would suggest is put those to the default, see 
> if they work, then (I'm assuming you had a reason for ding this) 
> find an acceptable port between 8000-.  


I think you get that backwards. Programs listening on port<1024 
will require root access. Anything larger than that, it doesn't
require special priviledge.


> If that doesn't work then we have a couple more "generic" 
> possiblities (such as running out of file handles which is
> very common with tomcat on solaris).  I wish I still had access to a 
> solaris box to try
> this..  The
> only  other difference was that I was using Jdk 1.3 (Sun).  Which I'd
> recommend for
> server side stuff unless you have a compelling reason not to. 
>  Its more
> stable, has
> less glitches and is faster.  Jdk 1.2.x was suns "wowee" and 
> jdk 1.3.x is a
> performance
> and stability release (IMHO)..  
> 
> If you haven't tweaked your file handles at any point then 
> there is a good
> chance that
> is in the way, unfortunately I do not remember what the 
> strings were to fix
> this, but
> I'm sure their documented somewhere (and they're not intuitive)..  


Thanks for the tips. We can't switch JDK for this release we're
having. But next release, we are hoping to use JDK 1.3


shinta

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[oliver2, andy]

 BTW we should redirect this to the tomcat-users list as I don't think
its a development issue.

I don't see anything wrong, and the only drastic difference from my previous
setup was that I had used the default 8007 and 8009.  Now This is purely
a guess, but I seem to remember reading that ports >1 were special
somehow
and required root access for some functions.  What I would suggest is put
those
to the default, see if they work, then (I'm assuming you had a reason for
ding this)
find an acceptable port between 8000-.  If that doesn't work then we
have a couple
more "generic" possiblities (such as running out of file handles which is
very common
with tomcat on solaris).  I wish I still had access to a solaris box to try
this..  The
only  other difference was that I was using Jdk 1.3 (Sun).  Which I'd
recommend for
server side stuff unless you have a compelling reason not to.  Its more
stable, has
less glitches and is faster.  Jdk 1.2.x was suns "wowee" and jdk 1.3.x is a
performance
and stability release (IMHO)..  

If you haven't tweaked your file handles at any point then there is a good
chance that
is in the way, unfortunately I do not remember what the strings were to fix
this, but
I'm sure their documented somewhere (and they're not intuitive)..  

Hope this helps some.

-Andy

-Original Message-
From: Shinta Tjio
To: 'oliver2, andy'; '[EMAIL PROTECTED] '
Cc: ''Dan Milstein' '
Sent: 3/6/01 7:51 PM
Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown

Here's exactly what I did to reproduce the problem. 
Again, I'm running this on a Solaris 2.8 SunOS machine, 
using JDK1.2.2, Apache 1.3.14, Tomcat 3.2.1. 

1. Start Apache 
2. Start Tomcat 
3. Start hitting Apache with multiple requests, such as 
   /example/servlet/HelloWorldExample. Make sure there 
   are some connections opened from Apache to Tomcat. 
   To make sure, you can do netstat -a | grep  
   where  is the ajp13 port. It should show 
   some socket in ESTABLISHED state. 
4. While the connections are in the ESTABLISHED state 
   (this should be the state because ajp13 reuse 
   connections), shutdown Tomcat. Now you will notice 
   the same netstat will show some sockets in FIN_WAIT2 
   and CLOSE_WAIT state. 
5. Now restart Tomcat. 
6. Repeat step 3. You will get Internal Server Error, up 
   to the number of children Apache has. After that number, 
   the page will be served. 

Attached is the server.xml, workers.properties, httpd.conf. 
My test servlet is called /mytest/servlet/testServlet but 
I think you can try it with any kind of servlet. 

I have been trying to code up the retry I mentioned below. 
I think I got it working. I just need to clean up the code 
some more. 

shinta 

> -Original Message- 
> From: oliver2, andy [ mailto:[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]> ] 
> Sent: Tuesday, March 06, 2001 6:33 PM 
> To: 'Shinta Tjio '; '[EMAIL PROTECTED] ' 
> Cc: ''Dan Milstein' ' 
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown 
> 
> 
>  I was running recently with that exact same configuration but did not

> experience that problem.  Can you show some config files. 
> 
> -Andy 
> 
> -Original Message- 
> From: Shinta Tjio 
> To: [EMAIL PROTECTED] 
> Cc: 'Dan Milstein' 
> Sent: 3/6/01 7:01 PM 
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown 
> 
> I am using Tomcat 3.2.1, Apache 1.3.14, running on 
> Solaris 2.8, Sun machines. 
> 
> After various attempts of debugging this, I have 
> more information. 
> 
> 1. Even though I'm setting the worker's property 
> cache_size to default (1), I'm finding there 
> are up to 6 connections opened from Apache to 
> Tomcat. I deduce this by looking at the mod_jk.conf 
> and by doing netstat. 
> 
> I finally find out, this is so because my Apache 
> is set to spawn minimum of 6 children and each 
> of those children are making separate connections 
> to Tomcat. 
> 
> This is very bad because, I ended up having to 
> reload 6 times before Tomcat starts serving me 
> the page again. Each time it uses a different 
> Apache children that has defunct socket. So the 
> more Apache children I have, the longer it takes 
> me to recover from this problem. 
> 
> 2. It seems when Tomcat dies & restarts, the send() 
> called by ajp13's jk_tcp_socket_sendfull() does not 
> get an error. But the recv() does get an error, with 
> errno ECONNRESET. After which, the socket is properly 
> closed. 
> 
> 3. When I shutdown Tomcat, those sockets that were 
> opened between Apache/Tomcat showed up in state 
> CLOSE_WAIT, and FIN_WAIT2. I think this is normally 
> solved by calling the shutdown() 

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





Here's exactly what I did to reproduce the problem.
Again, I'm running this on a Solaris 2.8 SunOS machine,
using JDK1.2.2, Apache 1.3.14, Tomcat 3.2.1.


1. Start Apache
2. Start Tomcat
3. Start hitting Apache with multiple requests, such as 
   /example/servlet/HelloWorldExample. Make sure there
   are some connections opened from Apache to Tomcat.
   To make sure, you can do netstat -a | grep 
   where  is the ajp13 port. It should show
   some socket in ESTABLISHED state.
4. While the connections are in the ESTABLISHED state
   (this should be the state because ajp13 reuse
   connections), shutdown Tomcat. Now you will notice
   the same netstat will show some sockets in FIN_WAIT2 
   and CLOSE_WAIT state.
5. Now restart Tomcat.
6. Repeat step 3. You will get Internal Server Error, up
   to the number of children Apache has. After that number, 
   the page will be served.


Attached is the server.xml, workers.properties, httpd.conf.
My test servlet is called /mytest/servlet/testServlet but
I think you can try it with any kind of servlet.


I have been trying to code up the retry I mentioned below.
I think I got it working. I just need to clean up the code
some more. 


shinta


> -Original Message-
> From: oliver2, andy [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 06, 2001 6:33 PM
> To: 'Shinta Tjio '; '[EMAIL PROTECTED] '
> Cc: ''Dan Milstein' '
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
> 
> 
>  I was running recently with that exact same configuration but did not
> experience that problem.  Can you show some config files.
> 
> -Andy
> 
> -Original Message-
> From: Shinta Tjio
> To: [EMAIL PROTECTED]
> Cc: 'Dan Milstein'
> Sent: 3/6/01 7:01 PM
> Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
> 
> I am using Tomcat 3.2.1, Apache 1.3.14, running on 
> Solaris 2.8, Sun machines. 
> 
> After various attempts of debugging this, I have 
> more information. 
> 
> 1. Even though I'm setting the worker's property 
> cache_size to default (1), I'm finding there 
> are up to 6 connections opened from Apache to 
> Tomcat. I deduce this by looking at the mod_jk.conf 
> and by doing netstat. 
> 
> I finally find out, this is so because my Apache 
> is set to spawn minimum of 6 children and each 
> of those children are making separate connections 
> to Tomcat. 
> 
> This is very bad because, I ended up having to 
> reload 6 times before Tomcat starts serving me 
> the page again. Each time it uses a different 
> Apache children that has defunct socket. So the 
> more Apache children I have, the longer it takes 
> me to recover from this problem. 
> 
> 2. It seems when Tomcat dies & restarts, the send() 
> called by ajp13's jk_tcp_socket_sendfull() does not 
> get an error. But the recv() does get an error, with 
> errno ECONNRESET. After which, the socket is properly 
> closed. 
> 
> 3. When I shutdown Tomcat, those sockets that were 
> opened between Apache/Tomcat showed up in state 
> CLOSE_WAIT, and FIN_WAIT2. I think this is normally 
> solved by calling the shutdown() API after closing 
> the socket. However, this would have to be done from 
> the Tomcat side in Ajp13ConnectionHandler.java. 
> I can't find the corresponding method of Socket 
> in Java. 
> 
> So.. based on all of these, the only fix I can think 
> of putting is to make mod_jk retry the send() if 
> recv() comes back with an error ECONNRESET. The retry 
> should happen after the old socket is properly closed. 
> 
> Anyone wants to comment? 
> 
> shinta 
>   
> 
> > -Original Message- 
> > From: Dan Milstein [ mailto:[EMAIL PROTECTED] 
> <mailto:[EMAIL PROTECTED]>
> ] 
> > Sent: Tuesday, March 06, 2001 12:00 PM 
> > To: [EMAIL PROTECTED] 
> > Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown 
> > 
> > 
> > What version of TC are you using?  What version of Apache? 
> > 
> > I would look into the mod_jk docs -- I think this is the 
> > spec'd behavior 
> > (which, admittedly, is not great, but that makes it more of 
> a feature 
> > request than a bug ;-).  With ajp13, Apache opens up a 
> > persistent TCP/IP 
> > connection TC -- if TC restarts, I think that connection may 
> > just hang up 
> > and then timeout (since Apache doesn't know that TC has restarted). 
> > 
> > If anyone wants to work on this, you would have the undying 
> > thanks of the 
> > rest of the TC community -- having to restart Apache all the 
> > time bugs a 
> > *lot* of people. 
> > 
&g

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[oliver2, andy]

 I was running recently with that exact same configuration but did not
experience that problem.  Can you show some config files.

-Andy

-Original Message-
From: Shinta Tjio
To: [EMAIL PROTECTED]
Cc: 'Dan Milstein'
Sent: 3/6/01 7:01 PM
Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown

I am using Tomcat 3.2.1, Apache 1.3.14, running on 
Solaris 2.8, Sun machines. 

After various attempts of debugging this, I have 
more information. 

1. Even though I'm setting the worker's property 
cache_size to default (1), I'm finding there 
are up to 6 connections opened from Apache to 
Tomcat. I deduce this by looking at the mod_jk.conf 
and by doing netstat. 

I finally find out, this is so because my Apache 
is set to spawn minimum of 6 children and each 
of those children are making separate connections 
to Tomcat. 

This is very bad because, I ended up having to 
reload 6 times before Tomcat starts serving me 
the page again. Each time it uses a different 
Apache children that has defunct socket. So the 
more Apache children I have, the longer it takes 
me to recover from this problem. 

2. It seems when Tomcat dies & restarts, the send() 
called by ajp13's jk_tcp_socket_sendfull() does not 
get an error. But the recv() does get an error, with 
errno ECONNRESET. After which, the socket is properly 
closed. 

3. When I shutdown Tomcat, those sockets that were 
opened between Apache/Tomcat showed up in state 
CLOSE_WAIT, and FIN_WAIT2. I think this is normally 
solved by calling the shutdown() API after closing 
the socket. However, this would have to be done from 
the Tomcat side in Ajp13ConnectionHandler.java. 
I can't find the corresponding method of Socket 
in Java. 

So.. based on all of these, the only fix I can think 
of putting is to make mod_jk retry the send() if 
recv() comes back with an error ECONNRESET. The retry 
should happen after the old socket is properly closed. 

Anyone wants to comment? 

shinta 
  

> -Original Message- 
> From: Dan Milstein [ mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
] 
> Sent: Tuesday, March 06, 2001 12:00 PM 
> To: [EMAIL PROTECTED] 
> Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown 
> 
> 
> What version of TC are you using?  What version of Apache? 
> 
> I would look into the mod_jk docs -- I think this is the 
> spec'd behavior 
> (which, admittedly, is not great, but that makes it more of a feature 
> request than a bug ;-).  With ajp13, Apache opens up a 
> persistent TCP/IP 
> connection TC -- if TC restarts, I think that connection may 
> just hang up 
> and then timeout (since Apache doesn't know that TC has restarted). 
> 
> If anyone wants to work on this, you would have the undying 
> thanks of the 
> rest of the TC community -- having to restart Apache all the 
> time bugs a 
> *lot* of people. 
> 
> -Dan 
> 
> > Shinta Tjio wrote: 
> > 
> > I'm having problem with mod_jk if ajp13 is used. 
> > 
> > The problem is often reproduced when Tomcat is shut 
> > down without Apache being shut down. When a request 
> > is fired through Apache as soon as Tomcat starts, 
> > I often get Internal Server Error. The mod_jk.log 
> > will have the following: 
> > 
> > > [jk_uri_worker_map.c (344)]: Into 
> jk_uri_worker_map_t::map_uri_to_worker 
> > 
> > > [jk_uri_worker_map.c (406)]: 
> jk_uri_worker_map_t::map_uri_to_worker, 
> > >   Found a match ajp13 
> > > [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 
> > > [jk_worker.c (127)]: wc_get_worker_for_name, done  found a worker 
> > > [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint 
> > > [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service 
> > > [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb 
> > > [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done 
> > > [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - 
> > >jk_tcp_socket_recvfull failed 
> > > [jk_ajp13_worker.c (619)]: Error reading request 
> > > [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done 
> > 
> > If I hit reload multiple times, eventually Tomcat will 
> > serve the servlet fine. 
> > 
> > Did anyone see this problem before? Is there anyway 
> > around this? 
> > 
> > shinta 
> 
> -- 
> 
> Dan Milstein // [EMAIL PROTECTED] 
> 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

RE: FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown





I am using Tomcat 3.2.1, Apache 1.3.14, running on
Solaris 2.8, Sun machines.


After various attempts of debugging this, I have 
more information.


1. Even though I'm setting the worker's property
cache_size to default (1), I'm finding there
are up to 6 connections opened from Apache to
Tomcat. I deduce this by looking at the mod_jk.conf
and by doing netstat. 


I finally find out, this is so because my Apache
is set to spawn minimum of 6 children and each
of those children are making separate connections
to Tomcat.


This is very bad because, I ended up having to
reload 6 times before Tomcat starts serving me
the page again. Each time it uses a different 
Apache children that has defunct socket. So the
more Apache children I have, the longer it takes
me to recover from this problem.


2. It seems when Tomcat dies & restarts, the send()
called by ajp13's jk_tcp_socket_sendfull() does not
get an error. But the recv() does get an error, with
errno ECONNRESET. After which, the socket is properly
closed.


3. When I shutdown Tomcat, those sockets that were
opened between Apache/Tomcat showed up in state 
CLOSE_WAIT, and FIN_WAIT2. I think this is normally
solved by calling the shutdown() API after closing
the socket. However, this would have to be done from
the Tomcat side in Ajp13ConnectionHandler.java. 
I can't find the corresponding method of Socket 
in Java. 


So.. based on all of these, the only fix I can think
of putting is to make mod_jk retry the send() if
recv() comes back with an error ECONNRESET. The retry
should happen after the old socket is properly closed.


Anyone wants to comment?


shinta
 


> -Original Message-
> From: Dan Milstein [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 06, 2001 12:00 PM
> To: [EMAIL PROTECTED]
> Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown
> 
> 
> What version of TC are you using?  What version of Apache?
> 
> I would look into the mod_jk docs -- I think this is the 
> spec'd behavior
> (which, admittedly, is not great, but that makes it more of a feature
> request than a bug ;-).  With ajp13, Apache opens up a 
> persistent TCP/IP
> connection TC -- if TC restarts, I think that connection may 
> just hang up
> and then timeout (since Apache doesn't know that TC has restarted).
> 
> If anyone wants to work on this, you would have the undying 
> thanks of the
> rest of the TC community -- having to restart Apache all the 
> time bugs a
> *lot* of people.
> 
> -Dan
> 
> > Shinta Tjio wrote:
> > 
> > I'm having problem with mod_jk if ajp13 is used.
> > 
> > The problem is often reproduced when Tomcat is shut
> > down without Apache being shut down. When a request
> > is fired through Apache as soon as Tomcat starts,
> > I often get Internal Server Error. The mod_jk.log
> > will have the following:
> > 
> > > [jk_uri_worker_map.c (344)]: Into 
> jk_uri_worker_map_t::map_uri_to_worker
> > 
> > > [jk_uri_worker_map.c (406)]: 
> jk_uri_worker_map_t::map_uri_to_worker,
> > >   Found a match ajp13
> > > [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13
> > > [jk_worker.c (127)]: wc_get_worker_for_name, done  found a worker
> > > [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint
> > > [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service
> > > [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb
> > > [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done
> > > [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error -
> > >    jk_tcp_socket_recvfull failed
> > > [jk_ajp13_worker.c (619)]: Error reading request
> > > [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done
> > 
> > If I hit reload multiple times, eventually Tomcat will
> > serve the servlet fine.
> > 
> > Did anyone see this problem before? Is there anyway
> > around this?
> > 
> > shinta
> 
> -- 
> 
> Dan Milstein // [EMAIL PROTECTED]
> 

Re: FW: problem w/ ajp13 - if Tomcat is shutdown

[Dan Milstein]

What version of TC are you using?  What version of Apache?

I would look into the mod_jk docs -- I think this is the spec'd behavior
(which, admittedly, is not great, but that makes it more of a feature
request than a bug ;-).  With ajp13, Apache opens up a persistent TCP/IP
connection TC -- if TC restarts, I think that connection may just hang up
and then timeout (since Apache doesn't know that TC has restarted).

If anyone wants to work on this, you would have the undying thanks of the
rest of the TC community -- having to restart Apache all the time bugs a
*lot* of people.

-Dan

> Shinta Tjio wrote:
> 
> I'm having problem with mod_jk if ajp13 is used.
> 
> The problem is often reproduced when Tomcat is shut
> down without Apache being shut down. When a request
> is fired through Apache as soon as Tomcat starts,
> I often get Internal Server Error. The mod_jk.log
> will have the following:
> 
> > [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker
> 
> > [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker,
> >   Found a match ajp13
> > [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13
> > [jk_worker.c (127)]: wc_get_worker_for_name, done  found a worker
> > [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint
> > [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service
> > [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb
> > [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done
> > [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error -
> >jk_tcp_socket_recvfull failed
> > [jk_ajp13_worker.c (619)]: Error reading request
> > [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done
> 
> If I hit reload multiple times, eventually Tomcat will
> serve the servlet fine.
> 
> Did anyone see this problem before? Is there anyway
> around this?
> 
> shinta

-- 

Dan Milstein // [EMAIL PROTECTED]

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]

FW: problem w/ ajp13 - if Tomcat is shutdown

[Shinta Tjio]

Title: FW: problem w/ ajp13 - if Tomcat is shutdown





I'm having problem with mod_jk if ajp13 is used.


The problem is often reproduced when Tomcat is shut
down without Apache being shut down. When a request 
is fired through Apache as soon as Tomcat starts, 
I often get Internal Server Error. The mod_jk.log
will have the following:


> [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker
> [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker,
>   Found a match ajp13
> [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13
> [jk_worker.c (127)]: wc_get_worker_for_name, done  found a worker
> [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint
> [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service
> [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb
> [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done
> [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - 
>    jk_tcp_socket_recvfull failed
> [jk_ajp13_worker.c (619)]: Error reading request
> [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done


If I hit reload multiple times, eventually Tomcat will
serve the servlet fine. 


Did anyone see this problem before? Is there anyway
around this?


shinta