Tomcat5.exe from 5.5.20 is half the size of 5.5.17, any change log to read?

2006-11-16 Thread Marziou, Gael
Hello,

Tomcat5.exe from 5.5.20 is half the size of 5.5.17 but I can't find any
change mentioned in Tomcat's change log page.

Is there a place to find out what has changed?

Thanks,

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Tomcat5.exe from 5.5.20 is half the size of 5.5.17, any change log to read?

2006-11-16 Thread Marziou, Gael

Thanks.

Do you know if it's actively supported? 

I submitted a bug report 2 months ago (maybe in the worng place) and got
no feedback.

http://issues.apache.org/jira/browse/DAEMON-86

I also would like to submit an enhancement request to be able to set the
processor affinity mask from the procrun monitor GUI.

Best regards,
Gael 




-Original Message-
From: Mladen Turk [mailto:[EMAIL PROTECTED] 
Sent: Thursday, November 16, 2006 6:43 PM
To: Tomcat Users List
Subject: Re: Tomcat5.exe from 5.5.20 is half the size of 5.5.17, any
change log to read?

Marziou, Gael wrote:
> Hello,
> 
> Tomcat5.exe from 5.5.20 is half the size of 5.5.17 but I can't find 
> any change mentioned in Tomcat's change log page.
> 
> Is there a place to find out what has changed?
> 

There is no change.
The latest build used dynamically linked MSVCRT, while the previous was
having that statically linked.
That's why the size is much smaller
(uses system msvcrt.dll)

Regards,
Mladen.

-
To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe,
e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Multi processor issue

2006-12-08 Thread Marziou, Gael
Hi,

As a preamble, let me say that I am not complaining about Tomcat (as the
problem root cause is uncertain) and I don't expect much assistance as
we are happy with our workaround.
However, our experience could help others or someone may have other
ideas.

We have recently switched a stable web application from JRun 4 to Tomcat
5.5.20 on Windows 2003 + Sun's JRE 1.5.0_08, running some testing for
months, but when we moved to production environment we immediately
experienced errors IllegalStateException: No modifications are allowed
to a locked ParameterMap.

Stack trace analysis concluded to RequestFacade objects shared between 2
threads.

We made code reviews of our application code and Tomcat's code and did
not find any thread safety issues. 

So, we checked all differences between staging and production, we
thought that the major difference was that the staging server was a
single CPU machine while the production servers had 2 physical
processors and 4 logical ones.

So, we constrained the Tomcat service to run on only one processor by
setting its processor affinity mask to 1 (run on CPU 0) and after
several weeks of monitoring we concluded that it "fixed" the problem.

So, we used a small utility (search for setaffinity.zip) to patch the
affinity mask in the Tomcat service executable (tomcat5.exe).

Later on, we set up a second production environment also on bi-processor
servers but we did not experience the same issue.
The hardware is not exactly the same as first production setup and the
OS is also different (Windows 2000 on 2nd env, Windows 2003 on 1st env
having the issue).

So, I don't believe it's a Tomcat, I would rather think to an H/W, or
JRE or OS issue.
In any case, it's a rather complex issue to investigate and we haven't
enough time for this as we are happy with performances running on one
single CPU.

Thanks,

Gael Marziou
Hewlett Packard

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
Thanks Chuck for your reply, I only cited part of your message where I
could follow up.
 
> Within Tomcat, request objects are normally processed by a single
thread 
> at a time, yet you note that two threads are accessing this particular

> request - where did the second one come from?  

Both were web threads and error did occur at the very beginning of
request processing when analyzing the request or when logging it in a
filter. 

> Is your app starting extra threads?  
> (Perhaps unknowingly, by using some third-party
> package.)

No. We checked this under profiler and all our threads are built by our
thread factories and do not manipulate any request objects only our own
business objects.

> Since requests are typically handled by a single thread, 
> there is little or no synchronization done during manipulation 
> of references associated with them, such as the parameter map.

That's also our understanding.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael

> could you tell me 32-bit or 64-bit OS, service pack levels, and what
the hardware is?  

Peter, I'm not sure it will help you, the only advice I can give you is
to test on same configuration as your future production environment. 
We knew it from the beginning but were not listened.

Windows 2003 Server Enterprise Edition SP1 (32 bits)
HP PROLIANT DL380 G3 4 x Intel Xeon 3.2 Ghz 

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
> Can you post the stack trace for that IllegalStateException?

Sure but that's only one of the symptoms we got.
I don't remember if the stack traces below were taken on 5.5.18 or
5.5.20 after we upgraded but as far as I know this part of the code was
not changed.


java.lang.IllegalStateException: No modifications are allowed to a
locked ParameterMap
at
org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165)
at
org.apache.catalina.connector.Request.getParameterMap(Request.java:1024)
at
org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad
e.java:411)
at
org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat
ionHttpRequest.java:727)
at
org.apache.catalina.core.ApplicationHttpRequest.getParameterMap(Applicat
ionHttpRequest.java:373)
at com.hp.swift.proxy.Proxy.fillParameters(Proxy.java:951)

And another one:

java.lang.IllegalStateException: No modifications are allowed to a
locked ParameterMap
at
org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165)
at
org.apache.catalina.connector.Request.getParameterMap(Request.java:1024)
at
org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad
e.java:411)
at
org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat
ionHttpRequest.java:727)
at
org.apache.catalina.core.ApplicationHttpRequest.getParameter(Application
HttpRequest.java:352)
at
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
This is our code and there's no threading here. 

-Original Message-
From: Caldarale, Charles R [mailto:[EMAIL PROTECTED] 
Sent: Friday, December 08, 2006 6:31 PM
To: Tomcat Users List
Subject: RE: Multi processor issue

> From: Marziou, Gael [mailto:[EMAIL PROTECTED]
> Subject: RE: Multi processor issue
> 
> com.hp.swift.portal.state.HttpEventContextObject.getParameter(
> HttpEventContextObject.java:64)

This looks suspicious - perhaps this portal code is the source of the
second thread.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe,
e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael
First, to clarify in my message, I did put 2 stack traces of 2
independent errors.

Tomcat does not synchronize before locking the map but that should not
be an issue because the request is guaranteed to be processed by only
one thread.

HttpEventContextObject.getParameter() does just extract the parameters
of the request to store them in what we cal an event, I suppose it could
be compared to a Struts action.
So, we only read the Request parameters, nothing fancy.


Gael

-Original Message-
From: Leon Rosenberg [mailto:[EMAIL PROTECTED] 
Sent: Friday, December 08, 2006 6:38 PM
To: Tomcat Users List
Subject: Re: Multi processor issue

looks like
Request.getParameterMap()
is called twice. The first call locks the request, the second fails.


what exactly does your
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)  ?

Could it be that two threads passes through this codepiece concurrently
and both calls comes through to the Request object? That would be a
clear synchronization issue and pretty typical for tomcat
:-)

regards
Leon

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-08 Thread Marziou, Gael

> Does the full stack trace show where the threads originated?  
> You appear to have given us only a partial of each error.

OK, see below.

> Is the request object stored in any session or global structure?

No.

> Could another thread have found it there and picked it up erroneously?


No.

> Do you have any listeners configured that might be getting in the
game?

No.


java.lang.IllegalStateException: No modifications are allowed to a
locked ParameterMap
at
org.apache.catalina.util.ParameterMap.put(ParameterMap.java:165)
at
org.apache.catalina.connector.Request.getParameterMap(Request.java:1024)
at
org.apache.catalina.connector.RequestFacade.getParameterMap(RequestFacad
e.java:411)
at
org.apache.catalina.core.ApplicationHttpRequest.parseParameters(Applicat
ionHttpRequest.java:727)
at
org.apache.catalina.core.ApplicationHttpRequest.getParameter(Application
HttpRequest.java:352)
at
com.hp.swift.portal.state.HttpEventContextObject.getParameter(HttpEventC
ontextObject.java:64)
at
com.hp.cii.radix.user.web.home.Home2Context.doUpdateSolutionSummary(Home
2Context.java:833)
at
com.hp.cii.radix.user.web.home.Home2Context.doDisplaySolutionSummary(Hom
e2Context.java:919)
at sun.reflect.GeneratedMethodAccessor593.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.hp.swift.util.MethodInvoker.invoke(MethodInvoker.java:52)
at
com.hp.swift.state.core.StateContextAction.execute(StateContextAction.ja
va:51)
at
com.hp.swift.state.core.Transition.executeAction(Transition.java:124)
at
com.hp.swift.state.core.StateMachineController.fireTransition(StateMachi
neController.java:1013)
at
com.hp.swift.state.core.StateMachineController.doHandle(StateMachineCont
roller.java:788)
at
com.hp.swift.state.core.StateMachineController.handle(StateMachineContro
ller.java:520)
at
com.hp.swift.portal.servlet.SmvController.doHandleRequest(SmvController.
java:393)
at
com.hp.swift.portal.servlet.SmvController.doGet(SmvController.java:235)
at
com.hp.swift.portal.servlet.SmvController.doPost(SmvController.java:215)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:359)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:301)
at
com.hp.swift.proxy.ProxySessionController.forwardRequest(ProxySessionCon
troller.java:57)
at
com.hp.swift.portal.servlet.SessionController$1.run(SessionController.ja
va:159)
at
com.hp.swift.security.manager.LightSecurityManager.doAsPrivileged(LightS
ecurityManager.java:272)
at
com.hp.swift.security.manager.SecurityChecker.doAsPrivileged(SecurityChe
cker.java:187)
at
com.hp.swift.portal.servlet.SessionController.forwardRequestAs(SessionCo
ntroller.java:155)
at
com.hp.swift.portal.servlet.SessionController.doHttp(SessionController.j
ava:132)
at
com.hp.swift.portal.servlet.SessionController.doPost(SessionController.j
ava:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at
com.hp.swift.portal.init.FirstLevelInitableServlet.service(FirstLevelIni
tableServlet.java:154)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
com.hp.swift.portal.servlet.HttpTracingFilter.doFilter(HttpTracingFilter
.java:89)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
com.hp.swift.portal.servlet.SecurityFilter.doFilter(SecurityFilter.java:
61)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:178)
at
org.apac

RE: Multi processor issue

2006-12-08 Thread Marziou, Gael

> But the evidence seems to show two threads are manipulating the same 
> request, otherwise you wouldn't get the error on the ParameterMap.  
> Does a full thread dump show any threads that aren't part of the set 
> started by Tomcat?

> either that, or a Request is reused before recycle() has been executed
completely.

We assumed this as well but after instrumenting more our code, we
concluded it was not possible because the corruption seemed to occur
before the end of the 2 requests processing and so before recycle() was
called.

I got some logs which sort of showed this but I can't find them right
now, I have stopped working on this issue 3 weeks ago.

> Alas, you could do a test and check whether the method getParameter is

> actually ever called by two threads during one lifecycle (patching the
> ParameterMap class, storing last accessing thread, and checking
whether
> another thread is already stored on access)

We did it but for some reason the log statements did not get logged in
our usual log file.
Then after we tried the affinity trick which made the symptom to
disappear and got no chance to try again.

> And Gael, the CPU is the most valuable resource in the production
environment, 
> are you really ready to give up your scaleability but simply not
investigating 
> further?

Well, this application is load balanced on 4 servers and the load is
currently pretty low on each of them even when running on a single CPU.
We did add more logs to our application and we did also instrument
Tomcat but with no luck and as we were unable to reproduce it in a dev
environment, it's difficult to ask for short downtimes to restart a
server in production.
On top of that, my team is being moved to another project...

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-11 Thread Marziou, Gael
Thanks for all replies, you have convinced me that we should investigate
further.

I propose to implement a bug catcher in Tomcat ParameterMap by storing
the thread that called the constructor amd then check its invariance
when calling other methods (e.g. isLocked) of this class, if the
invariance is violated we will throw an exception.

It seems to me that with this data, we should be able to determine which
code is accessing the map and so we should know whether it's a bug in
the application or not.

Any other suggestion?


Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Marziou, Gael
Dong, I'm not sure I understand your suggestion, I attached below my bug
catcher in org.apache.catalina.connector.Request.

Do you mean I should add a delay before the while loop to increase the
length of the critical section?

What do you mean by yelding the CPU (sorry I'm French) ?

Gael

public Map getParameterMap() {

synchronized(parameterMap) {
Thread currentThread = Thread.currentThread();
if (pmapThread == null) {
// Store context of first caller thread
pmapThread = currentThread;
pmapStackTrace = currentThread.getStackTrace();
} else {
if (currentThread != pmapThread) {
String msg = "Original thread = " + pmapThread +
"\nOriginal stack trace = " +
stack2string(pmapStackTrace) +
"\nCurrent thread = " + currentThread;
throw new IllegalStateException(msg);
}
}
}

if (parameterMap.isLocked())
return parameterMap;

Enumeration enumeration = getParameterNames();
while (enumeration.hasMoreElements()) {
String name = enumeration.nextElement().toString();
String[] values = getParameterValues(name);
parameterMap.put(name, values);
}

parameterMap.setLocked(true);

return parameterMap;

} 

-Original Message-
From: JiaDong Huang [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, December 12, 2006 10:42 AM
To: 'Tomcat Users List'
Subject: RE: Multi processor issue

Gael,

If you can build the Tomcat code/component, put in time delay code and
also yield the CPU, within the ParameterMap initialization procedure.

Dong

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-12 Thread Marziou, Gael
I used sleep() as you can see in the modified version of
getParameterMap() at the end of this message.

I got the issue and here is the trace.

What I call "original thread" is the one that was the first to call
Requuest.getParameterMap()

Correct me if I am wrong but it seems to me that the 2 threads are from
Tomcat, not from our application.

Gael


java.lang.IllegalStateException: 

Original thread = Thread[http-80-3,5,main]
Original stack trace =  java.lang.Thread.dumpThreads()
java.lang.Thread.getStackTrace()

org.apache.catalina.connector.Request.getParameterMap()(Request.java:102
3)

org.apache.catalina.connector.RequestFacade.getParameterMap()(RequestFac
ade.java:411)

org.apache.catalina.core.ApplicationHttpRequest.parseParameters()(Applic
ationHttpRequest.java:727)

org.apache.catalina.core.ApplicationHttpRequest.getParameter()(Applicati
onHttpRequest.java:352)

com.hp.swift.portal.state.HttpEventContextObject.getParameter()(HttpEven
tContextObject.java:64)

com.hp.cii.radix.user.web.home.Home2Context.doUpdateAlertSummary()(Home2
Context.java:709)

com.hp.cii.radix.user.web.home.Home2Context.doDisplayAlertSummary()(Home
2Context.java:789)
sun.reflect.NativeMethodAccessorImpl.invoke0()
sun.reflect.NativeMethodAccessorImpl.invoke()
sun.reflect.DelegatingMethodAccessorImpl.invoke()
java.lang.reflect.Method.invoke()
com.hp.swift.util.MethodInvoker.invoke()(MethodInvoker.java:52)

com.hp.swift.state.core.StateContextAction.execute()(StateContextAction.
java:51)

com.hp.swift.state.core.Transition.executeAction()(Transition.java:124)

com.hp.swift.state.core.StateMachineController.fireTransition()(StateMac
hineController.java:1013)

com.hp.swift.state.core.StateMachineController.doHandle()(StateMachineCo
ntroller.java:788)

com.hp.swift.state.core.StateMachineController.handle()(StateMachineCont
roller.java:520)

com.hp.swift.portal.servlet.SmvController.doHandleRequest()(SmvControlle
r.java:393)

com.hp.swift.portal.servlet.SmvController.doGet()(SmvController.java:235
)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:689)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:802)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:252)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

org.apache.catalina.core.ApplicationDispatcher.invoke()(ApplicationDispa
tcher.java:672)

org.apache.catalina.core.ApplicationDispatcher.processRequest()(Applicat
ionDispatcher.java:463)

org.apache.catalina.core.ApplicationDispatcher.doForward()(ApplicationDi
spatcher.java:359)

org.apache.catalina.core.ApplicationDispatcher.forward()(ApplicationDisp
atcher.java:301)

com.hp.swift.proxy.ProxySessionController.forwardRequest()(ProxySessionC
ontroller.java:57)

com.hp.swift.portal.servlet.SessionController$1.run()(SessionController.
java:164)

com.hp.swift.security.manager.LightSecurityManager.doAsPrivileged()(Ligh
tSecurityManager.java:272)

com.hp.swift.security.manager.SecurityChecker.doAsPrivileged()(SecurityC
hecker.java:187)

com.hp.swift.portal.servlet.SessionController.forwardRequestAs()(Session
Controller.java:159)

com.hp.swift.portal.servlet.SessionController.doHttp()(SessionController
.java:134)

com.hp.swift.portal.servlet.SessionController.doGet()(SessionController.
java:110)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:689)

com.hp.swift.portal.init.FirstLevelInitableServlet.service()(FirstLevelI
nitableServlet.java:154)
javax.servlet.http.HttpServlet.service()(HttpServlet.java:802)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:252)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

com.hp.swift.portal.servlet.HttpTracingFilter.doFilter()(HttpTracingFilt
er.java:94)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:202)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

com.hp.swift.portal.servlet.SecurityFilter.doFilter()(SecurityFilter.jav
a:61)

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter()(Appli
cationFilterChain.java:202)

org.apache.catalina.core.ApplicationFilterChain.doFilter()(ApplicationFi
lterChain.java:173)

org.apache.catalina.core.StandardWrapperValve.invoke()(StandardWrapperVa
lve.java:213)

org.apache.catalina.core.StandardContextValve.invoke()(StandardContextVa
lve.java:178)

org.apache.catalina.core.StandardHostValve.invoke()(StandardHostValve.ja
va:126)

org.apache.catalina.valv

RE: Multi processor issue

2006-12-12 Thread Marziou, Gael

> Anything dealing with a session must be careful not to expose a
> Request object to the session scope, since there's the risk another
> request for the same session may get confused with the first - I
> suspect that's what's happening here. 

Will do.

> You should take a careful look at all the com.hp code in the stack
> trace (not just the above three places) and verify that Request
> objects are not getting mixed up.  Might want to add some trace code
> in various places to insure that a given thread is procesing the same
> Request object at all points. 

Yes, it makes sense.

> Could also try turning off APR (remove or rename tcnative-1.dll) and
> see if that has an effect. 

I already did at the beginning of our investigation as I suspected the
JNI code but it had no effect.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-20 Thread Marziou, Gael
Some fresh news and hopefully I have narrowed the scope enough to get
the root cause.

First, the previous traces that I posted last week were wrong because my
bug catcher was buggy: I forgot to reset the fields in recycle().

So, I fixed this and saved few other fields in
org.apache.catalina.connector.Request in getParameterMap(), we got
another trace and found that we needed also to log the association of
thread and request at different steps in our application in methods that
were found in the stack traces.

So, we ran again the bug catcher and this time, it failed with the 3rd
symptom which did not involve getParameterMap() but still was due to a
request object being shared between 2 http threads.

Hopefully, the mailing list supports attachments as I have attached the
log file.
I edited the log file to remove statements from other sessions using
other threads and also user data (e.g. email address).

So, here is my analysis of what happened.

Line 1: We get a request for "red.gif" on thread http-80-5

Line 2: We get a request for "Green.gif" on thread http-80-4

Then you can follow up different threads either by using thread name or
by using log4j NDC which consists in jsessionid/request# (e.g.
D1D4F7EE6827BC3B569FE2F3BE3C0F15/4).

Line 9 and 10: We can see that each thread has still its own
RequestFacade object with a different URI still consistent with lines 1
and 2. The log statement is done at ProxySessionController.java:52

Line 11 and 12: here is the problem the 2 threads share the same
ApplicationHttpRequest object and same request URI.

So now let's take a look at the stack trace by concatenating the stack
of the cause and its nesting exception.


67  at
com.hp.swift.proxy.passthrough.PassThroughController.doHttp(PassThroughC
ontroller.java:100)
68  at
com.hp.swift.proxy.passthrough.PassThroughController.doGet(PassThroughCo
ntroller.java:77)
69  at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
70  at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
71  at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:252) 

18  at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:275)
19  at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:173)
20  at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:672)
21  at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:463)
22  at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:359)
23  at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:301)
24  at
com.hp.swift.proxy.ProxySessionController.forwardRequest(ProxySessionCon
troller.java:60)


So, we see that the problems occurs between
ProxySessionController.forwardRequest() and the first line of
PassThroughController.doHttp(). In fact we can go down to
PassThroughController.doGet() as it is just directly calling doHttp()
without any change.

At this point, I must explain what does
ProxySessionController.forwardRequest(): it does select a
RequestDispatcher (a servlet) based on the request path info and then
calls forward() on it.
An important point here is that we re-use the same RequestDispatcher got
once by ServletContext.getNamedDispatcher() over and over.
In Tomcat, the class implementing RequestDispatcher is
org.apache.catalina.core.ApplicationDispatcher

So, it means that the 2 concurrent threads will call forward() on same
ApplicationDispatcher object.

By looking at the code of ApplicationDispatcher, I would tend to think
that the answer is no because of fields like wrapRequest and requestURI
(there are other fields but they are unused) however the javadoc does
not mention it.

Looking at org.apache.catalina.core.ApplicationContextFacade and
org.apache.catalina.core.ApplicationContext, it seems to me that
getNamedDispatcher() always return a fresh RequestDispatcher object, so
it seems that it is intended to be used only once per request.

Thanks for your help,

Gael 



 1  2006-12-20 07:23:04,257: INFO  
com.hp.swift.portal.servlet.HttpTracingFilter - Starting GET 
/radix/bin/infoboard/images/red.gif HTTP/1.1 {http-80-5 | [EMAIL PROTECTED] 
 2  2006-12-20 07:23:04,257: INFO  
com.hp.swift.portal.servlet.HttpTracingFilter - Starting GET 
/radix/bin/infoboard/images/Green.gif HTTP/1.1 {http-80-4 | [EMAIL PROTECTED] 
 3  2006-12-20 07:23:04,257: INFO  com.hp.swift.portal.servlet.request - 
(D1D4F7EE6827BC3B569FE2F3BE3C0F15/4) Starting GET 
/radix/bin/infoboard/images/red.gif HTTP/1.1
 4  2006-12-20 07:23:04,257: INFO  com.hp.swift.portal.servlet.request - 
(D1D4F7EE6827BC3B569FE2F3BE3C0F15/5) Starting GET 
/radix/bin/infoboard/images/Green.gif HTTP/1.1
 5  2006-12-20 07:23:04,257: INFO  log.appli - 
(D1D4F7EE6827BC3B569FE2F3BE3C0F15/4) SessionCont

RE: Multi processor issue

2006-12-20 Thread Marziou, Gael
> By looking at the code of ApplicationDispatcher, I would tend to think
that 
> the answer is no because of fields like wrapRequest and requestURI
(there
> are other fields but they are unused) however the javadoc does not
mention it.

I forgot to include the question but some of you may have guessed it ;-)
The question was: can we call forward() on same ApplicationDispatcher
object from 2 concurrent threads?

Gael 




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-21 Thread Marziou, Gael
> Mark Thomas said:
> 
> I have re-read the sections of the spec you quoted several 
> times and I don't see anything that suggests to me that the 
> RequestDispatcher has only request level scope that would 
> suggest the OPs use is invalid.

That's the key point, also I am under the impression that the
ApplicationDispatcher could be re-factored to be thread safe by avoiding
having request scope fields and using only local variables in forward()
method.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-21 Thread Marziou, Gael
> I don't think this is a bug in TC's implementation. This is a 
> relatively subtle mistake that seems both easy enough to make 
> and easy to fix.

Maybe Tomcat should enforce this like for instance when we get an
exception trying to write into a response that was already committed.
It would have saved weeks of effort on our side.

> Gael, what was the reasoning behind caching that 
> RequestDispatcher? Was it simply because you would have been 
> calling the same method with the same parameters repeatedly, 
> and figured that you'd get some performance edge by saving 
> the resulting object? Were you observing any performance 
> degradation at any point? Or was this a case of premature 
> optimization? ;)

We did implement this 2 years ago, so details have vanished but it was
mainly based on our spec interpretation and also I admit on premature
optimization.

In several places, the spec defines different behaviors of
RequestDispatcher when it was obtained using getNamedDispatcher(String)
and noticeably for handling parameters.
So, our interpretation was that a RequestDispatcher obtained using
getNamedDispatcher() was different than one obtained using
getRequestDispatcher(): it has a different behavior and a wider scope.
This was the reason behind our implementation and also the fact that our
previous container (before migrating to Tomcat) seemed to work this way
(as far as I remember).

> I'd really like to hear if simply eliminating the caching of 
> this object fixes your problem. Does it?

So far so good, the new code has been running fine for 2 hours while it
was failing usually within 30 minutes.

Gael

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Multi processor issue

2006-12-22 Thread Marziou, Gael

> Can you suggest a fix? I'm not sure how this kind of thing 
> could be safely veto'd... for instance, it might actually be 
> appropriate for a RequestDispatcher to be re-used in some 
> context (say, repeating a request twice) or even to do so 
> with different threads, as long as the access is done 
> strictly serially.

Do you mean submittingan enhancement request with some proposed code?

> Fantastic! I'm glad we were able to help, even if it did take a while.

Yes and I want to thank all the people from the mailing list for their
good advices and also for giving me additional motivation to investigate
further.

Gael 

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]