Re: high CPU usage on tomcat 7
Aren't you're clients polling the server, this may cause the session not to expire even if the user is not using the UI ? Jeff On Sat, Oct 6, 2012 at 9:42 PM, Mark Thomas ma...@apache.org wrote: Kirill Kireyev kir...@instagrok.com wrote: Thanks for all your thorough advice Shanti! (and everyone else) Here are my findings so far: 0) My servlet sessions store a large number (~10s of M) or data in RAM. This is by design, to optimize performance. I can also have ~3K active sessions at any one time. Hence a large heap size. 1) When I (1) manually expire inactive sessions through Tomcat web interface and (2) manually hit Perform GC through jvisualvm console attached to the tomcat process, everything works great, and the memory is successfully reclaimed, and the used heap size drops back to what it should be when the application initializes. 2) However, this doesn't seem to work automatically. More specifically: a) Sessions are not expiring without manually doing it (I can see the number growing in the Tomcat web interface). Even though my conf/web.xml says: session-config session-timeout20/session-timeout /session-config b) A full garbage collection is not being performed unless I do it manually. I'm attaching the GC records from my logs/catalina.out. Any insights? Session expiration is performed by the background processing thread. What is that thread doing (a thread dump - well several over time - will tell you). Fix the session expiration issue and the GC issue will be solved. Note the JVM may well not perform a full GC unless it absolutely has to. In fact, the JVM may not perform any GC if it doesn't have to. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
RE: high CPU usage on tomcat 7
Le 4 oct. 2012 14:38, Caldarale, Charles R chuck.caldar...@unisys.com a écrit : From: Kirill Kireyev [mailto:kir...@instagrok.com] Subject: Re: high CPU usage on tomcat 7 Perhaps what I need is to have the JVM do garbage collections more frequently, so that they don't become a huge CPU-hogging ordeals when they do happen. That's not how it works. The amount of time spent in a GC operation is dependent almost entirely on the number of live objects in the heap, not the amount of space being reclaimed. Having a smaller heap might improve your cache hit rate, but it will have little effect on the time used by GC. It is highly likely that your webapps have a memory leak. You need to use a profiler and find out what's eating up all that memory. If you set the heap size to a smaller value, you should be able to provoke the situation earlier, but it certainly won't fix it. +1 Jeff - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: high CPU usage on tomcat 7
I don't think a cpu loop will make tomcat stopping responding to requests
I will make it very slow to respond
But a shortage on memory is hard to recover
Jeff
Le vendredi 28 septembre 2012, mailingl...@j-b-s.de a écrit :
Maybe an infinite loop? We observed something similar due to a bug in the
java regex impl and certain user input causes this regex looping behaviour.
As this locked one core but to the user it simply looked like server was
not responding, guess what happend? Right: they press refresh page and next
core gone :-)
So running state might be evil, too.
Switch on GC logging, maybe its just related to a full gc. In case this
happens again take a thread dump and search similarities. Take multiple
dumps in a 5 sec interval. Try to find long running threads (in our case we
noticed the regex bug)
Jens
Sent from my iPhone
On 27.09.2012, at 22:05, Kirill Kireyev kir...@instagrok.com wrote:
Thanks for all the advice everyone! There is a possibility that the CPU
is caused by an app thread - I am looking into that possibility. Will let
you know when I find out more.
Thanks,
Kirill
On 9/27/12 12:17 PM, Shanti Suresh wrote:
Hi Kirill,
Like Mark, Bill and Jeff said, those threads are normal
request-processing
threads. I have included a script that might help with isolating high
CPU
issues with Tomcat.
Also, I think it might be helpful to see how the Java heap is
performing as
well.
Please bring up Jconsole and let it run over the week. Inspect the
graphs
for Memory, CPU and threads. Since you say that high CPU occurs
intermittently several times during the week and clears itself, I
wonder if
it is somehow related with the garbage collection options you are using
for
the server. Or it may be a code-related problem.
Things to look at may include:
(1) Are high CPU times related to Java heap reductions happening at the
same time? == GC possibly needs tuning
(2) Are high CPU times related to increase in thread usage? ==
possible
livelock in looping code?
(3) how many network connections come into the Tomcat server during
high-CPU times?Possible overload-related?
Here is the script. I made a couple of small changes, for e.g.,
changing
the username. But didn't test it after the change. During high-CPU
times,
invoke the script a few times, say 30 seconds apart. And then compare
the
thread-dumps. I like to use TDA for thread-dump analysis of Tomcat
thread-dumps.
Mark, et al, please feel free to help me refine this script. I would
like
to have a script to catch STUCK threads too :-) Let me know if anyone
has
a script already. Thanks.
--high_cpu_diagnostics.pl:-
#!/usr/bin/perl
#
use Cwd;
# Make a dated directory for capturing current diagnostics
my ($sec,$min,$hour,$mday,$mon,$year,
$wday,$yday,$isdst) = localtime time;
$year += 1900;
$mon += 1;
my $pwd = cwd();
my $preview_diag_dir =
/tmp/Preview_Diag.$year-$mon-$mday-$hour:$min:$sec;
print $preview_diag_dir\n;
mkdir $preview_diag_dir, 0755;
chdir($preview_diag_dir) or die Can't chdir into $preview_diag_dir
$!\n;
# Capture Preview thread dump
my $process_pattern = preview;
my $preview_pid = `/usr/bin/pgrep -f $process_pattern`;
my $login = getpwuid($) ;
if (kill 0, $preview_pid){
#Possible to send a signal to the Preview Tomcat - either
webinf
or root
my $count = kill 3, $preview_pid;
}else {
# Not possible to send a signal to the VCM - use sudo
system (/usr/bin/sudo /bin/kill -3 $preview_pid);
}
# Capture Preview thread dump
system (/usr/bin/jmap
-dump:format=b,file=$preview_diag_dir/preview_heapdump.hprof
$preview_pid);
# Gather the top threads; keep around for reference on what other
threads
are running
@top_cmd = (/usr/bin/top, -H, -n1, -b);
@sort_cmd = (/bin/sort, -r, -n, -k, 9,9);
@sed_cmd = (/bin/sed, -n, '8,$p');
system(@top_cmd 1 top_all_threads.log);
# Get your tomcat user's threads, i.e. threads of user, webinf
system('/usr/bin/tail -n+6 top_all_threads.log | /bin/sort -r -n -k
9,9 |
/bin/grep webinf top_all_threads.log 1 top_user_webinf_threads.log');
# Get the thread dump instaGrok_sml.png
--
Jeff MAURY
Legacy code often differs from its suggested alternative by actually
working and scaling.
- Bjarne Stroustrup
http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.twitter.com/jeffmaury
Re: high CPU usage on tomcat 7
This is probably due to out of memory, I have the same problem on my ubuntu ci machine Did you monitor your tomcat with jmx ? Jeff Le 27 sept. 2012 17:39, Kirill Kireyev kir...@instagrok.com a écrit : Hi! I'm periodically getting unduly high (100%) CPU usage by the tomcat process on my server. This problems happens intermittently, several times a week. When the server goes into this high CPU it does not come back (and becomes unresponsive to new requests), and the only recourse is to restart the tomcat process. I'm using Tomcat 7.0.30, with APR and Apache web server, on a Ubuntu 11.10 server with 32g of RAM / 8 CPUs. I've done several jstack stack traces when this occurs, and what I consistently see, are the connector threads in the RUNNABLE state every time, i.e.: ajp-apr-8009-Acceptor-0 daemon prio=10 tid=0x010a1000 nid=0x539 runnable [0x7f9364f8e000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.**accept(Native Method) at org.apache.tomcat.util.net.**AprEndpoint$Acceptor.run(** AprEndpoint.java:1013) at java.lang.Thread.run(Thread.**java:722) http-apr-8443-Acceptor-0 daemon prio=10 tid=0x0109b800 nid=0x535 runnable [0x7f936551] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.**accept(Native Method) at org.apache.tomcat.util.net.**AprEndpoint$Acceptor.run(** AprEndpoint.java:1013) at java.lang.Thread.run(Thread.**java:722) http-apr-8080-Acceptor-0 daemon prio=10 tid=0x015ab000 nid=0x531 runnable [0x7f9365a92000] java.lang.Thread.State: RUNNABLE at org.apache.tomcat.jni.Socket.**accept(Native Method) at org.apache.tomcat.util.net.**AprEndpoint$Acceptor.run(** AprEndpoint.java:1013) at java.lang.Thread.run(Thread.**java:722) Other threads are in RUNNBLE too in different cases, but these are the one that are always there when the high CPU occurs. That's why I'm starting to think it has something to do with Tomcat. Can anyone shed some light on this? My current Connector configurations in server.xml are: Connector port=8080 protocol=org.apache.coyote.** http11.Http11AprProtocol connectionTimeout=2 maxThreads=500 minSpareThreads=10 maxSpareThreads=20 redirectPort=8443 pollTime=10 / ... Connector port=8443 protocol=org.apache.coyote.** http11.Http11AprProtocol maxThreads=200 scheme=https secure=true SSLEnabled=true SSLCACertificateFile=** SSLCertificateKeyFile=** SSLCertificateFile=*** enableLookups=false clientAuth=false sslProtocol=TLS pollTime=10 / ... Connector port=8009 protocol=AJP/1.3 redirectPort=8443 acceptCount=100 connectionTimeout=5000 keepAliveTimeout=2 disableUploadTimeout=true enableLookups=false maxHttpHeaderSize=8192 maxSpareThreads=75 maxThreads=150 minSpareThreads=25 executor=default / Thanks a lot! -Kirill --**--**- To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.orgusers-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Issue building the Unix daemon (jsvc) with the configure script in Tomcat7
of permission-errors grief login tomcat mkdir /datadisk1/tomcat/conf/Catalina/localhost BTW when you do a ls -al /datadisk1/tomcat/conf/Catalina/localhost you should have god rights rwd pretty much everything sudo is tricky because you'll need to add tomcat user to run the tomcat daemon(sh) to /etc/sudoers but then you'll only have permission to run daemon(.sh) access to temp, work and log folders have not been granted cd $CATALINA_BASE chown tomcat work chown tomcat logs chown tomcat temp Your clock is 6 hours ahead of me so you will receive this email very early am..i assume you're in CET I'll check back SAT roughly 8am CST Martin Gainty __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. From: fja...@gmail.com Date: Sat, 25 Aug 2012 00:22:50 +0200 Subject: Re: Issue building the Unix daemon (jsvc) with the configure script in Tomcat7 To: users@tomcat.apache.org Hi Jeff, I tried both: [ec2-user@ip-10-34-89-185 bin]$ ./daemon.sh start Cannot set group id for user 'tomcat' Error validating user 'tomcat' With sudo, I get the permissions issue. ec2-user is a sudoer. sudo ./daemon.sh start *SEVERE: Unable to create directory for deployment: /datadisk1/tomcat/conf/Catalina/localhost* Aug 24, 2012 1:29:12 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /datadisk1/tomcat/webapps/examples Aug 24, 2012 1:29:12 PM org.apache.catalina.core.StandardContext postWorkDirectory *WARNING: Failed to create work directory [/datadisk1/tomcat/work/Catalina/localhost/examples] for context [/examples] * Aug 24, 2012 1:29:13 PM org.apache.jasper.EmbeddedServletOptions init *SEVERE: The scratchDir you specified: /datadisk1/tomcat/work/Catalina/localhost/examples is unusable.* I am trying to see the whole command passed to jsvc but I am not fluent in shell scripts and I can't get to echo it yet. Fred On Fri, Aug 24, 2012 at 10:45 PM, Jeff MAURY jeffma...@jeffmaury.com wrote: It probably means your current user (ec2-user) has not the rights to run as tomcat user. Try the same command with sudo in front, if ec2-user is part of the sudoers Jeff On Fri, Aug 24, 2012 at 4:09 PM, Fred Janon fja...@gmail.com wrote: Ah! Thanks Jeff, you are right, thanks. I checked and the OpenJDK installed is not the full JDK (??? confusing). I had to install the OpenJDK-dev to get the full JDK. I managed to build the jsvc (yeah!), now I am getting an error after starting tomcat using daemon.sh, it seems that it runs a root, not tomcat as I configured setenv.sh. tomcat cannot create some dirs, reported in catalina-daemon.out. If I launch tomcat without being root, I get this error: [ec2-user@ip-10-34-89-185 bin]$ ./daemon.sh start Cannot set group id for user 'tomcat' Error validating user 'tomcat' Not sure what this means, I have a tomcat user in the tomcat group. If I launch tomcat using that command: sudo ./daemon.sh start with setenv.sh: export CATALINA_HOME=/datadisk1/tomcat export CATALINA_OPTS=-server -Xms128M -Xmx512M export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre export TOMCAT_USER=tomcat then catalina-daemon.out shows: Aug 24, 2012 1:29:11 PM org.apache.catalina.core.AprLifecycleListener init SEVERE: An incompatible version 1.1.22 of the APR based Apache Tomcat Native library
Re: Issue building the Unix daemon (jsvc) with the configure script in Tomcat7
Fred, you are getting confused by the term OpenJDK. It comes in two flavor, JRE and JDK. I have some .h files on my Ubuntu server, but not under /usr/lib/jvm/java-1.6.0-openjdk but under /usr/lib/jvm/java-6-openjdk Regards Jeff On Fri, Aug 24, 2012 at 8:52 AM, Fred Janon fja...@gmail.com wrote: Hi Jeff, Thanks, but I read the instructions and requirements for a C compiler an the JDK. As shown in my original message there is a JDK /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/* but no jni_md.h in the Open JDK. As I said the jni_md.h seems to be in the Windows JDK, but not in the Linux one. Cheers, Fred On Fri, Aug 24, 2012 at 7:51 AM, Jeff MAURY jeffma...@jeffmaury.com wrote: You need a JDK to be installed on your machine and it seems you've only a JRE Jeff On Thu, Aug 23, 2012 at 10:12 PM, Fred Janon fja...@gmail.com wrote: Hi, I am trying to build the Unix deamon jsvc in AWS EC2 Linux. I did not install Tomcat7 that comes with the AWS Linux, I downloaded Tomcat7 from the Apache Tomcat website, I installed it and ran Tomcat7 successfully. Now I am trying to build the daemon to run Tomcat7 as a daemon. I get an error running the configure script as indicated in the Tomcat documentation. I did a lot of searching on the web and cannot find a solution to this issue. I am not fluent in shell scripting, so my understanding of configure is very limited. Do I need to regenerate the configure script or do I need to modify the script? The error is: *checking for JDK os include directory... Cannot find jni_md.h in /usr/lib/jvm/jre/* *configure: error: You should retry --with-os-type=SUBDIR* * * It looks like *jni_md.h* is part of the Windows JDK, *but it's not in the Linux OpenJDK on CentOS.* I don't really understand why the message says *with-os-type* where it seems to mean JDK? As indicated in the INSTALL.txt below, I am reporting the issue, hoping that someone can give me a solution or a patch for the script... Depending on your JDK layout, configure might fail to find the JNI machine dependant include file (jni_md.h). If that's the case use the --with-os-type=subdir parameter where subdir points to the directory within JDK include directory containing jni_md.h file. If your operating system is supported, configure will go thru cleanly, otherwise it will report an error (please send us the details of your OS/JDK, or a patch against the sources). Below is the console output running configure and some information about the jdk installed. [ec2-user@ip-10-244-162-78 unix]$ *./configure* *** Current host *** checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking cached host system type... ok *** C-Language compilation tools *** checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ANSI C... none needed checking for ranlib... ranlib checking for strip... strip *** Host support *** checking C flags dependant on host system type... ok *** Java compilation tools *** *checking for JDK os include directory... Cannot find jni_md.h in /usr/lib/jvm/jre/* *configure: error: You should retry --with-os-type=SUBDIR* [ec2-user@ip-10-244-162-78 unix]$ find / -name 'jni_md.h' 2/dev/null [ec2-user@ip-10-244-162-78 unix]$ find / -name '*jni*' 2/dev/null /usr/share/java-jni /datadisk1/tomcat/bin/commons-daemon-1.0.10-native-src/windows/src/javajni.c /datadisk1/tomcat/bin/commons-daemon-1.0.10-native-src/windows/include/javajni.h Installed Packages java-1.6.0-openjdk.i686 installed [ec2-user@ip-10-244-162-78 unix]$ *./configure -with-java=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre* *** Current host *** checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking cached host system type... ok *** C-Language compilation tools *** checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ANSI C... none needed checking for ranlib... ranlib checking for strip... strip *** Host support *** checking C
Re: Issue building the Unix daemon (jsvc) with the configure script in Tomcat7
org.apache.jasper.EmbeddedServletOptions init SEVERE: The scratchDir you specified: /datadisk1/tomcat/work/Catalina/localhost/manager is unusable. Aug 24, 2012 1:29:13 PM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory /datadisk1/tomcat/webapps/host-manager Aug 24, 2012 1:29:13 PM org.apache.catalina.core.StandardContext postWorkDirectory WARNING: Failed to create work directory [/datadisk1/tomcat/work/Catalina/localhost/host-manager] for context [/host-manager] Aug 24, 2012 1:29:13 PM org.apache.jasper.EmbeddedServletOptions init SEVERE: The scratchDir you specified: /datadisk1/tomcat/work/Catalina/localhost/host-manager is unusable. Aug 24, 2012 1:29:13 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [http-bio-8080] Aug 24, 2012 1:29:13 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler [ajp-bio-8009] Aug 24, 2012 1:29:13 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 1613 ms I am not sure my export TOMCAT_USER=tomcat is right, or if I am missing something. Regards, Fred On Fri, Aug 24, 2012 at 12:01 PM, Jeff MAURY jeffma...@jeffmaury.com wrote: Fred, you are getting confused by the term OpenJDK. It comes in two flavor, JRE and JDK. I have some .h files on my Ubuntu server, but not under /usr/lib/jvm/java-1.6.0-openjdk but under /usr/lib/jvm/java-6-openjdk Regards Jeff On Fri, Aug 24, 2012 at 8:52 AM, Fred Janon fja...@gmail.com wrote: Hi Jeff, Thanks, but I read the instructions and requirements for a C compiler an the JDK. As shown in my original message there is a JDK /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/* but no jni_md.h in the Open JDK. As I said the jni_md.h seems to be in the Windows JDK, but not in the Linux one. Cheers, Fred On Fri, Aug 24, 2012 at 7:51 AM, Jeff MAURY jeffma...@jeffmaury.com wrote: You need a JDK to be installed on your machine and it seems you've only a JRE Jeff On Thu, Aug 23, 2012 at 10:12 PM, Fred Janon fja...@gmail.com wrote: Hi, I am trying to build the Unix deamon jsvc in AWS EC2 Linux. I did not install Tomcat7 that comes with the AWS Linux, I downloaded Tomcat7 from the Apache Tomcat website, I installed it and ran Tomcat7 successfully. Now I am trying to build the daemon to run Tomcat7 as a daemon. I get an error running the configure script as indicated in the Tomcat documentation. I did a lot of searching on the web and cannot find a solution to this issue. I am not fluent in shell scripting, so my understanding of configure is very limited. Do I need to regenerate the configure script or do I need to modify the script? The error is: *checking for JDK os include directory... Cannot find jni_md.h in /usr/lib/jvm/jre/* *configure: error: You should retry --with-os-type=SUBDIR* * * It looks like *jni_md.h* is part of the Windows JDK, *but it's not in the Linux OpenJDK on CentOS.* I don't really understand why the message says *with-os-type* where it seems to mean JDK? As indicated in the INSTALL.txt below, I am reporting the issue, hoping that someone can give me a solution or a patch for the script... Depending on your JDK layout, configure might fail to find the JNI machine dependant include file (jni_md.h). If that's the case use the --with-os-type=subdir parameter where subdir points to the directory within JDK include directory containing jni_md.h file. If your operating system is supported, configure will go thru cleanly, otherwise it will report an error (please send us the details of your OS/JDK, or a patch against the sources). Below is the console output running configure and some information about the jdk installed. [ec2-user@ip-10-244-162-78 unix]$ *./configure* *** Current host *** checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking cached host system type... ok *** C-Language compilation tools *** checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ANSI C... none needed checking for ranlib... ranlib checking for strip... strip *** Host support *** checking C flags dependant on host system type... ok *** Java compilation tools *** *checking for JDK os include
Re: Issue building the Unix daemon (jsvc) with the configure script in Tomcat7
: Cannot find jni_md.h in
${JAVA_HOME}/${OS} 5
echo ${ECHO_T}Cannot find jni_md.h in ${JAVA_HOME}/${OS} 6
{ { echo $as_me:$LINENO: error: You should retry
--with-os-type=SUBDIR 5
echo $as_me: error: You should retry --with-os-type=SUBDIR 2;}
{ (exit 1); exit 1; }; }
fi
fi
-
Thanks
Fred
--
Jeff MAURY
Legacy code often differs from its suggested alternative by actually
working and scaling.
- Bjarne Stroustrup
http://www.jeffmaury.com
http://riadiscuss.jeffmaury.com
http://www.twitter.com/jeffmaury
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
That what I guessed but I don't understand everything. The code you are referencing is related to NTPipes and not sockets. So I'm not familiar with Tomcat Native implementation, but do you know the global architecture ? What I guessed is that the Tomcat native stuff created with the inherit flag, but even if it does, I don't see why it should not work if the socket is closed after a timeout. Regards Jeff On Fri, Jun 29, 2012 at 11:22 AM, Laurent Petit lpe...@yseop.com wrote: Hello Jeff, Konstantin all, On Mon, 2012-06-25 at 20:52 +0200, verlag.preis...@t-online.de wrote: Hello Jeff all, Von: Jeff MAURY jeffma...@jeffmaury.com Datum: Mon, 25 Jun 2012 18:46:02 +0200 Konstantin, your explanations are very interesting but unclear to me: what do you call the inactivity timer ? When it is started ? After the request has been processed by the servlet ? In that case, I see no difference between a servlet that launch a process and another one. It seems to me that a process that is launched does not inhererits handles from its parent process but it's possible that under Windows, it's an option so it would be interesting to watch. Jeff Sorry, I'm just a normal Tomcat user, and I don't know how exactly the APR connector and its Timeout works, so I am unable to answer that. Howewer, I did some further observations: -When I perform a request to the servlet that opens wordpad.exe, the TCP connection from Tomcat does not close after the timeout - even when I kill the Tomcat process (java.exe), the TCP connection is still open. If I kill wordpad.exe, then finally the connection is closed/aborted. -When I have 1 TCP connection open to Tomcat and the servlet starts the little C program, Task manager shows that it has 11 handles. However, when I have 5 TCP connections open to Tomcat, and do the request on one of them, Task maanger shows that the C program has 15 handles - so four more handles when there are four more connections to Tomcat. All of that 5 TCP connections don't close until I kill that process. That seems to me to be an indication that socket handles could be inherited by the child processes that are startet by ProcessBuilder from tomcat. A msdn article mentions how to create a new process using CreateProcess(); it also mentions that socket handles can be inherited: http://msdn.microsoft.com/en-us/library/windows/desktop/ms724466.aspx However, as I don't have much knowledge about programming with WinAPIs, I don't know why those handles are inherited (the MSDN article mentions that a handle must be specified as inheritable when created, to allow a child process to inherit it). Maybe someone with more WinAPI/Tomcat Native knowledge can help here. I also had the vague intuition that this related to handles inheritence. Your recent tests research tend to make this hypothesis even more appealing. I know nothing about tomcat-native implementation, but I was able to see that the bInheritHandle member is set to true here in tomcat native's C code: lpetit:~/tmp/tomcat-native $ grep bInheritHandle -R * native/os/win32/ntpipe.c:con-sa.bInheritHandle = TRUE; ( http://svn.apache.org/repos/asf/tomcat/native/branches/1.1.x/native/os/win32/ntpipe.c) Jeff, What else could we do to help investigate / fix this issue ? Cheers, -- Laurent Regards, Konstantin Preißer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Laurent Petit Agence +33 (0)4 78 47 07 49 Email lpe...@yseop.com Yseop apporte une réponse intelligente et individualisée à chacun de vos clients www.yseop.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
No, it does not (yet) make sense because one piece of the puzzle is missing. The Local socket is not a TCP socket, it is created from a path rather than from an IP address. So this socket is in no way linked to the browser, at least directly, and the missing piece is what is the relation from this socket to the browser. Jeff On Fri, Jun 29, 2012 at 2:40 PM, Laurent Petit lpe...@yseop.com wrote: Hello, On Fri, 2012-06-29 at 11:45 +0200, Jeff MAURY wrote: That what I guessed but I don't understand everything. The code you are referencing is related to NTPipes and not sockets. I'm not a C expert, but my bet is that the file name is misleading. Indeed, the con-sa.bInheritHandle = TRUE; statement appears in the definition driven by the following macro expansion: TCN_IMPLEMENT_CALL(jlong, Local, create)(TCN_STDARGS, jstring name, jlong pool) And when the macro call TCN_IMPLEMENT_CALL(jlong, Local, create) is preprocessed, it gives: JNIEXPORT RT JNICALL Java_org_apache_tomcat_jni_##Local##_##create , thus the bInheritHandle = TRUE statement is defined in the following function signature: JNIEXPORT RT JNICALL Java_org_apache_tomcat_jni_##Local##_##create(TCN_STDARGS, jstring name, jlong pool) which is a function called by JNDI, and as I guess, by class org.apache.tomcat.ini.Local, method create() : http://svn.apache.org/repos/asf/tomcat/native/branches/1.1.x/java/org/apache/tomcat/jni/Local.java And the javadoc for class Local is: Local socket And for methode create : /** * Create a socket. * @param path The address of the new socket. * @param cont The parent pool to use * @return The new socket that has been set up. */ It seems like the native code uses a pool, and my bet is that the pool is responsible for trying to reuse the socket. And something like this may happen: the socket handle, inherited by the child process, puts the socket in such a state that the pool either can't, either doesn't want to reuse it. And then what happens is that on the other side of the socket, the client continues to write HTTP requests, and nobody is listening. And then when the Tomcat's child process is killed, the socket is finally closed somehow, the client browser notices it and creates a new connection to the server and retries the HTTP request. Does that make sense ? So I'm not familiar with Tomcat Native implementation, but do you know the global architecture ? What I guessed is that the Tomcat native stuff created with the inherit flag, but even if it does, I don't see why it should not work if the socket is closed after a timeout. Regards Jeff On Fri, Jun 29, 2012 at 11:22 AM, Laurent Petit lpe...@yseop.com wrote: Hello Jeff, Konstantin all, On Mon, 2012-06-25 at 20:52 +0200, verlag.preis...@t-online.de wrote: Hello Jeff all, Von: Jeff MAURY jeffma...@jeffmaury.com Datum: Mon, 25 Jun 2012 18:46:02 +0200 Konstantin, your explanations are very interesting but unclear to me: what do you call the inactivity timer ? When it is started ? After the request has been processed by the servlet ? In that case, I see no difference between a servlet that launch a process and another one. It seems to me that a process that is launched does not inhererits handles from its parent process but it's possible that under Windows, it's an option so it would be interesting to watch. Jeff Sorry, I'm just a normal Tomcat user, and I don't know how exactly the APR connector and its Timeout works, so I am unable to answer that. Howewer, I did some further observations: -When I perform a request to the servlet that opens wordpad.exe, the TCP connection from Tomcat does not close after the timeout - even when I kill the Tomcat process (java.exe), the TCP connection is still open. If I kill wordpad.exe, then finally the connection is closed/aborted. -When I have 1 TCP connection open to Tomcat and the servlet starts the little C program, Task manager shows that it has 11 handles. However, when I have 5 TCP connections open to Tomcat, and do the request on one of them, Task maanger shows that the C program has 15 handles - so four more handles when there are four more connections to Tomcat. All of that 5 TCP connections don't close until I kill that process. That seems to me to be an indication that socket handles could be inherited by the child processes that are startet by ProcessBuilder from tomcat. A msdn article mentions how to create a new process using CreateProcess(); it also mentions that socket handles can be inherited: http://msdn.microsoft.com/en-us/library/windows/desktop/ms724466.aspx However, as I don't have much knowledge about programming with WinAPIs
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
Hello, I found the correct code to look at: http://svn.apache.org/repos/asf/tomcat/native/branches/1.1.x/native/src/network.c It does not set the inherit flag but it is using the apr library that maybe set the flag under Windows. If this is the case and according to your references, it is likely the client will not receive the ack and this is a problem with APR and Runtime.exec Jeff On Fri, Jun 29, 2012 at 4:38 PM, Laurent Petit lpe...@yseop.com wrote: On Fri, 2012-06-29 at 15:27 +0200, Jeff MAURY wrote: No, it does not (yet) make sense because one piece of the puzzle is missing. The Local socket is not a TCP socket, it is created from a path rather than from an IP address. So this socket is in no way linked to the browser, at least directly, and the missing piece is what is the relation from this socket to the browser. You're right, so the question is still open. In the mean time, I'm come across interesting google search results, among which the first one seems really interesting: Oracle Bug 6428742 == Currently, on Windows, Runtime.exec is implemented by calling CreateProcess with the bInheritHandles argument set to TRUE, which causes open handles to not be closed if they are not explicitly set to be non-inherited. This is the underlying cause of many subtle bugs, such as 6347873: (so) Ports opened with ServerSocketChannel blocks when using Runtime.exec http://monaco.sfbay/detail.jsf?cr=6347873; == Note the if they are not explicitly set to be non-inherited = this seems like a game changer, since our implicit assumption was that we would just look for code which would have accidentally set the handle to be inherited, not the other way around. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6428742 QT Project == When using spawned processes, sockets opened by the client remain in CLOSE_WAIT state after closing the connection. Sockets on the remote side would be in FIN_WAIT_2 state and waiting for ACK. When the spawned (child) process is closed, then ACK is sent and sockets are finally moved to CLOSE state on both sides. This seems to be caused by the default handle inheritance on Windows by child processes. https://bugreports.qt-project.org/browse/QTBUG-4465 Zeromq2 project === On Windows platform, in order to prevent child processes to inherit parent sockets, SetHandleInformation ( http://msdn.microsoft.com/en-us/library/windows/desktop/ms724935(v=vs.85).aspx) should be called on the socket handle. This patch comes out from discussion at: https://github.com/zeromq/clrzmq/issues/60; https://github.com/zeromq/zeromq2-x/pull/51 and https://zeromq.jira.com/browse/LIBZMQ-366 I found way more interesting results, but I'll spare space and won't paste them all in this thread ( google search criteria : windows inheriting socket handles ) Jeff On Fri, Jun 29, 2012 at 2:40 PM, Laurent Petit lpe...@yseop.com wrote: Hello, On Fri, 2012-06-29 at 11:45 +0200, Jeff MAURY wrote: That what I guessed but I don't understand everything. The code you are referencing is related to NTPipes and not sockets. I'm not a C expert, but my bet is that the file name is misleading. Indeed, the con-sa.bInheritHandle = TRUE; statement appears in the definition driven by the following macro expansion: TCN_IMPLEMENT_CALL(jlong, Local, create)(TCN_STDARGS, jstring name, jlong pool) And when the macro call TCN_IMPLEMENT_CALL(jlong, Local, create) is preprocessed, it gives: JNIEXPORT RT JNICALL Java_org_apache_tomcat_jni_##Local##_##create , thus the bInheritHandle = TRUE statement is defined in the following function signature: JNIEXPORT RT JNICALL Java_org_apache_tomcat_jni_##Local##_##create(TCN_STDARGS, jstring name, jlong pool) which is a function called by JNDI, and as I guess, by class org.apache.tomcat.ini.Local, method create() : http://svn.apache.org/repos/asf/tomcat/native/branches/1.1.x/java/org/apache/tomcat/jni/Local.java And the javadoc for class Local is: Local socket And for methode create : /** * Create a socket. * @param path The address of the new socket. * @param cont The parent pool to use * @return The new socket that has been set up. */ It seems like the native code uses a pool, and my bet is that the pool is responsible for trying to reuse the socket. And something like this may happen: the socket handle, inherited by the child process, puts the socket in such a state that the pool either can't, either doesn't want to reuse it. And then what happens is that on the other side of the socket, the client continues to write HTTP requests, and nobody is listening. And then when the Tomcat's child process is killed
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
If you can post a Mavenized example, I could have time to run it and watch at it. Regards Jeff On Mon, Jun 25, 2012 at 11:40 AM, Laurent Petit lpe...@yseop.com wrote: Hello, Is there anything I can do to help qualify the problem even more ? Beyond having shown the source code of the servlet, would a fully mavenized example help? Anything else? Regards, -- Laurent On Fri, 2012-06-22 at 10:21 +0200, Laurent Petit wrote: Hello, On Thu, 2012-06-21 at 22:53 +0200, Jeff MAURY wrote: Is it specific to Wordpad or any launched process will do the trick ? Do you tried with a non UI process (console) ? I did an additional test, as you suggested, with a non UI process (a small java executable launched in headless mode, whose purpose was just to wait for 20 seconds before exiting). I can reproduce the problem with this headless executable: the HTTP client is blocked until the 20 seconds elapse and the process is killed. Here is the modified servlet code I used: https://www.refheap.com/paste/3285 and here is the code for the small java program: https://www.refheap.com/paste/3286 Regards, -- Laurent Jeff On Thu, Jun 21, 2012 at 10:45 PM, Laurent Petit lpe...@yseop.com wrote: En Réponse à Tomcat Users List users@tomcat.apache.org le 21 juin 2012 00:07 Can you verify your 2 threads (reading input an error) are launched ? Yes they are. Verified. Can you confirm you are getting the problem only on Windows ? Yes sir, we're unable to reproduce it on Ubuntu. Jeff On Wed, Jun 20, 2012 at 4:19 PM, Laurent Petit lpe...@yseop.com wrote: Hello, I have a problem with keep-alive connections, when starting a subprocess (via JDK's default ProcessBuilder/Process API), while also having started Tomcat with the APR HTTP/1.1 Connector. The problem symptom is with Keep-Alive connection, as follows: - the client (browser, jmeter, etc.) sends a first request - the servlet starts a Process for e.g. wordpad.exe - the servlet returns an acknowledgment html content, sets the content length, flushes the writer, and returns - the client displays the received acknowledgement html content, sends the second request to the server. - the server doesn't answer. No Tomcat log ever reports the start of something received. - Then when on the server you close the wordpad.exe instance, the server finally handles the second request. I have created a small servlet code which reproduces the problem. For the demo, it suffices to have the started subprocess be wordpad.exe. Of course my real process is more interesting than that :-). You can find the servlet code for reproducing the test here: https://www.refheap.com/paste/3254To reproduce the problem deterministically, the ConnectionTimeout for the APR HTTP Connector in servlet.xml must be set sufficiently low. With my boxes, I get a 100% error hit when set at 200 ms. You can find here the jmeter script which hits the same page again and again with keep-alive option set on: https://www.refheap.com/paste/3255So far, the only reliable solution we have found to work around this problem is to not use the HTP/1.1 APR Connector.My configuration : Windows 2008 US std R2 64bits Tomcat 6.0.32 64 bits Java 6u30 64 bits(Oracle) Same issue has been observed on Windows 7 32 bits. Was not able to reproduce the issue on Linux Ubuntu Desktop 11.10 or 12.04.Thanks for your support, ideas, solutions, etc. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Laurent Petit Agence +33 (0)4 78 47 07 49 Email lpe...@yseop.com Yseop apporte une réponse intelligente et individualisée à chacun de vos clients www.yseop.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
Yes, fine Jeff On Mon, Jun 25, 2012 at 1:46 PM, Laurent Petit lpe...@yseop.com wrote: On Mon, 2012-06-25 at 11:51 +0200, Jeff MAURY wrote: If you can post a Mavenized example, I could have time to run it and watch at it. Ok, thanks, I will. Is a mavenized project with source code published on github ok for you ? Regards Jeff On Mon, Jun 25, 2012 at 11:40 AM, Laurent Petit lpe...@yseop.com wrote: Hello, Is there anything I can do to help qualify the problem even more ? Beyond having shown the source code of the servlet, would a fully mavenized example help? Anything else? Regards, -- Laurent On Fri, 2012-06-22 at 10:21 +0200, Laurent Petit wrote: Hello, On Thu, 2012-06-21 at 22:53 +0200, Jeff MAURY wrote: Is it specific to Wordpad or any launched process will do the trick ? Do you tried with a non UI process (console) ? I did an additional test, as you suggested, with a non UI process (a small java executable launched in headless mode, whose purpose was just to wait for 20 seconds before exiting). I can reproduce the problem with this headless executable: the HTTP client is blocked until the 20 seconds elapse and the process is killed. Here is the modified servlet code I used: https://www.refheap.com/paste/3285 and here is the code for the small java program: https://www.refheap.com/paste/3286 Regards, -- Laurent Jeff On Thu, Jun 21, 2012 at 10:45 PM, Laurent Petit lpe...@yseop.com wrote: En Réponse à Tomcat Users List users@tomcat.apache.org le 21 juin 2012 00:07 Can you verify your 2 threads (reading input an error) are launched ? Yes they are. Verified. Can you confirm you are getting the problem only on Windows ? Yes sir, we're unable to reproduce it on Ubuntu. Jeff On Wed, Jun 20, 2012 at 4:19 PM, Laurent Petit lpe...@yseop.com wrote: Hello, I have a problem with keep-alive connections, when starting a subprocess (via JDK's default ProcessBuilder/Process API), while also having started Tomcat with the APR HTTP/1.1 Connector. The problem symptom is with Keep-Alive connection, as follows: - the client (browser, jmeter, etc.) sends a first request - the servlet starts a Process for e.g. wordpad.exe - the servlet returns an acknowledgment html content, sets the content length, flushes the writer, and returns - the client displays the received acknowledgement html content, sends the second request to the server. - the server doesn't answer. No Tomcat log ever reports the start of something received. - Then when on the server you close the wordpad.exe instance, the server finally handles the second request. I have created a small servlet code which reproduces the problem. For the demo, it suffices to have the started subprocess be wordpad.exe. Of course my real process is more interesting than that :-). You can find the servlet code for reproducing the test here: https://www.refheap.com/paste/3254To reproduce the problem deterministically, the ConnectionTimeout for the APR HTTP Connector in servlet.xml must be set sufficiently low. With my boxes, I get a 100% error hit when set at 200 ms. You can find here the jmeter script which hits the same page again and again with keep-alive option set on: https://www.refheap.com/paste/3255So far, the only reliable solution we have found to work around this problem is to not use the HTP/1.1 APR Connector.My configuration : Windows 2008 US std R2 64bits Tomcat 6.0.32 64 bits Java 6u30 64 bits(Oracle) Same issue has been observed on Windows 7 32 bits. Was not able to reproduce the issue on Linux Ubuntu Desktop 11.10 or 12.04.Thanks for your support, ideas, solutions, etc. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Laurent Petit Agence +33 (0)4 78 47 07 49 Email lpe...@yseop.com Yseop apporte une réponse intelligente et individualisée à chacun de vos clients www.yseop.com - To unsubscribe, e-mail: users
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
Konstantin, your explanations are very interesting but unclear to me: what do you call the inactivity timer ? When it is started ? After the request has been processed by the servlet ? In that case, I see no difference between a servlet that launch a process and another one. It seems to me that a process that is launched does not inhererits handles from its parent process but it's possible that under Windows, it's an option so it would be interesting to watch. Jeff On Mon, Jun 25, 2012 at 6:23 PM, verlag.preis...@t-online.de verlag.preis...@t-online.de wrote: Von: Laurent Petit lpe...@yseop.com Datum: Mon, 25 Jun 2012 17:46:06 +0200 Hello Jeff al, Here it is, I've created a Git repository with the sources for the webapp the test (maven projects), and instructions in the README on how to reproduce the issue. https://github.com/lpetit-yseop/apr-test Hope this will help you get started on reproducing the problem, Regards, Hello, I also could reproduce the problem on WIndows 7 64 Bit with Java 1.7.0_05 (64-bit) and Tomcat 7.0.28, using TC Native 1.1.24. I also used a small connectionTimeout=200. What I can see is, that with a normal request (that doesn't start a new process), Tomcat processes the request, and after an inactivity timeout of 200 ms, the APR connector closes the TCP connection. When the browser makes another request, it does open a new connection for that request. However, when a request goes to a servlet that starts a new process, and the timeout of 200 ms occurs, the TCP connection is not closed - but Tomcat doesn't seem to read on it any more, so further requests sent on that TCP connection will never be processed. When the new process (wordpad.exe) is closed, the TCP connection will be aborted. Then, the browser probably establishes a new TCP connection and re-sends it request there, which is processed by Tomcat. Unfortunately, don't have any knowledge of programming native (C) programms using Windows APIs, so I can only make a guess: Could it be that some handles to files/sockets are inherited to the child process (wordpad.exe) when launched by Java's ProcessBuilder, which prevent the socket from being closed? E.g., when I launch a simple C app (which calls WaitForSingleObject(-1, 99)) from the Windows explorer, I can see 7 handles in the taskmanger; whereas when it is launched by the servlet in tomcat, Task manager shows 11 handles - but as said, I don't know if that has something to do with the problem. Regards, Konstantin Preißer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Re : Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
Is it specific to Wordpad or any launched process will do the trick ? Do you tried with a non UI process (console) ? Jeff On Thu, Jun 21, 2012 at 10:45 PM, Laurent Petit lpe...@yseop.com wrote: En Réponse à Tomcat Users List users@tomcat.apache.org le 21 juin 2012 00:07 Can you verify your 2 threads (reading input an error) are launched ? Yes they are. Verified. Can you confirm you are getting the problem only on Windows ? Yes sir, we're unable to reproduce it on Ubuntu. Jeff On Wed, Jun 20, 2012 at 4:19 PM, Laurent Petit lpe...@yseop.com wrote: Hello, I have a problem with keep-alive connections, when starting a subprocess (via JDK's default ProcessBuilder/Process API), while also having started Tomcat with the APR HTTP/1.1 Connector. The problem symptom is with Keep-Alive connection, as follows: - the client (browser, jmeter, etc.) sends a first request - the servlet starts a Process for e.g. wordpad.exe - the servlet returns an acknowledgment html content, sets the content length, flushes the writer, and returns - the client displays the received acknowledgement html content, sends the second request to the server. - the server doesn't answer. No Tomcat log ever reports the start of something received. - Then when on the server you close the wordpad.exe instance, the server finally handles the second request. I have created a small servlet code which reproduces the problem. For the demo, it suffices to have the started subprocess be wordpad.exe. Of course my real process is more interesting than that :-). You can find the servlet code for reproducing the test here: https://www.refheap.com/paste/3254To reproduce the problem deterministically, the ConnectionTimeout for the APR HTTP Connector in servlet.xml must be set sufficiently low. With my boxes, I get a 100% error hit when set at 200 ms. You can find here the jmeter script which hits the same page again and again with keep-alive option set on: https://www.refheap.com/paste/3255So far, the only reliable solution we have found to work around this problem is to not use the HTP/1.1 APR Connector.My configuration : Windows 2008 US std R2 64bits Tomcat 6.0.32 64 bits Java 6u30 64 bits(Oracle) Same issue has been observed on Windows 7 32 bits. Was not able to reproduce the issue on Linux Ubuntu Desktop 11.10 or 12.04.Thanks for your support, ideas, solutions, etc. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Creating separate log for each request
Are you sure of that, a single file for a single HTTP request. I don't know a file system capable to handle that ? Jeff On Wed, Jun 20, 2012 at 10:38 PM, Lahiru Gunathilake glah...@gmail.comwrote: Hi All, I have my webapp deployed on tomcat. I am using slf4j for my logging and everything works fine and all the logs comes to catalina.out. But I have a requirement of creating a separate log for each request comes to my webapp... I only have single webapp, so I simply want to create a separate log for each http request come to my tomcat instance. Anybody knows how to do this in tomcat configuration ? Regards Lahiru -- System Analyst Programmer PTI Lab Indiana University -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Issue with keep-alive connections, when using APR Connector on Windows and starting Processes from Servlets
Can you verify your 2 threads (reading input an error) are launched ? Can you confirm you are getting the problem only on Windows ? Jeff On Wed, Jun 20, 2012 at 4:19 PM, Laurent Petit lpe...@yseop.com wrote: Hello, I have a problem with keep-alive connections, when starting a subprocess (via JDK's default ProcessBuilder/Process API), while also having started Tomcat with the APR HTTP/1.1 Connector. The problem symptom is with Keep-Alive connection, as follows: - the client (browser, jmeter, etc.) sends a first request - the servlet starts a Process for e.g. wordpad.exe - the servlet returns an acknowledgment html content, sets the content length, flushes the writer, and returns - the client displays the received acknowledgement html content, sends the second request to the server. - the server doesn't answer. No Tomcat log ever reports the start of something received. - Then when on the server you close the wordpad.exe instance, the server finally handles the second request. I have created a small servlet code which reproduces the problem. For the demo, it suffices to have the started subprocess be wordpad.exe. Of course my real process is more interesting than that :-). You can find the servlet code for reproducing the test here: https://www.refheap.com/paste/3254 To reproduce the problem deterministically, the ConnectionTimeout for the APR HTTP Connector in servlet.xml must be set sufficiently low. With my boxes, I get a 100% error hit when set at 200 ms. You can find here the jmeter script which hits the same page again and again with keep-alive option set on: https://www.refheap.com/paste/3255 So far, the only reliable solution we have found to work around this problem is to not use the HTP/1.1 APR Connector. My configuration : Windows 2008 US std R2 64bits Tomcat 6.0.32 64 bits Java 6u30 64 bits(Oracle) Same issue has been observed on Windows 7 32 bits. Was not able to reproduce the issue on Linux Ubuntu Desktop 11.10 or 12.04. Thanks for your support, ideas, solutions, etc. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: tomcat7:run with multi maven modules changes
You need to write a M2E connector for a Maven MOJO that processes your projects files. I don't think this is the case for the tomcat7:run goal and it should work out of of the box in Eclipse. Just need to add a Maven lunch configuration with the proper goal. Regards Jeff On Tue, May 8, 2012 at 5:34 PM, Olivier Lamy ol...@apache.org wrote: I imagine you use tomcat7:run inside eclipse. Do that work outside eclipse ? Perso I don't have any issues as I use only command line and furthermore I don't use eclipse :-). I imagine there is some m2e connector to write 2012/5/8 Albert Kam moonblade.w...@gmail.com: I notice that i have to repeat these steps so that tomcat7:run will see the newest stuffs from other maven modules : - rebuild all of my mvn modules (including my webapp module on which my tomcat7:run-ed, and other dependencies module) - clean my eclipse java project, rebuilding everything - tomcat7:run will then see the newest stuffs Note that i dont need to rebuild maven and eclipse project if i dont modify the webapp maven module, tomcat7 can automatically see the newest changes Ideally i would like to skip rebuilding everything in maven and then eclipse to see the newest changes in the other modules outside webapp module. Are there any tips to achieve this ? Thanks ! -- Do not pursue the past. Do not lose yourself in the future. The past no longer is. The future has not yet come. Looking deeply at life as it is in the very here and now, the practitioner dwells in stability and freedom. (Thich Nhat Hanh) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Does Tomcat Maven plugin support WAR overlays ?
Salut Olivier, I wanted to make sure I understand the goal properly. Is the rationale for using src/main/webapp as the default docbase for Tomcat to propagate modifications of files on the fly (JSP,...) without the need to restart Tomcat ? At least, I think we should make it clear in the documentation and issue a warning at runtime when we detect an overlay. I will open a JIRA Jeff On Sat, Apr 28, 2012 at 11:38 PM, Olivier Lamy ol...@apache.org wrote: Salut Jeff, Current Overlay support with tomcat6/7:run is very limited (only use jars from WEB-INF/lib of the the war dependencies). Perso, I use maven-dependency-plugin to extract war content (see sample in this pom [1] ). I agree it's hackhish :-) and having a better support as in the war plugin could be better. But didn't yet have any time to work on that. (can you create an issue for that ?) As my goal was to cut a release soon (ideally starting release process next week, I'm not sure I will have time to work on that) -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy [1] http://svn.apache.org/repos/asf/archiva/trunk/archiva-modules/archiva-web/archiva-webapp/pom.xml 2012/4/28 Jeff MAURY jeffma...@jeffmaury.com: Hello, I am facing the following problem with WAR overlays: I have a first WAR, called skeleton, that contains all necessary stuff: base web.xml, index.jsp and JAR dependencies. I have another WAR whose first dependency is the skeleton which is (as of yet) almost empty except for the slf4j_log4 dependency. When i run tomcat7:run on the skeleton, everything is ok When I run tomcat7:run on the second war, Tomcat start but I am not able to use the application. I have look at the mojo code and it seems it is using the web app source directory (src/main/webapp) by default so as it is empty my case, it cannot work. AM I missing something ? Thanks Jeff -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Re: Does Tomcat Maven plugin support WAR overlays ?
I don't link the idea of having Maven adding some stuff except under target. This will cause many many problems with people and SCM. I don't get your second solution: how do you merge you temporary stuff and src/main/webapp with a symlink ? Jeff On Mon, Apr 30, 2012 at 4:52 PM, Olivier Lamy ol...@apache.org wrote: 2012/4/30 Jeff MAURY jeffma...@jeffmaury.com: Salut Olivier, I wanted to make sure I understand the goal properly. Is the rationale for using src/main/webapp as the default docbase for Tomcat to propagate modifications of files on the fly (JSP,...) without the need to restart Tomcat ? Yup that's the goal. Fast dev mode (i.e. no restart) when only modifying jsp or static resources (html, js, css etc...) At least, I think we should make it clear in the documentation and issue a warning at runtime when we detect an overlay. What I can do is to extract the war content (except WEB-INF/lib as it's already added) under the warSourceDirectory plugin parameter (default src/main/webapp) which is the docBase (but with a parameter called extractPath relative to the docBase). The plugin configuration part could be overlays overlay groupIdorg.foo/groupId artifactIdbar/artifactId extractPathbar/extractPath /overlay /overlays What I don't like is people will have to ignore this directory in their scm The best: extract somewhere (temporary directory) and add this as a symlink in the docBase. I will open a JIRA Jeff On Sat, Apr 28, 2012 at 11:38 PM, Olivier Lamy ol...@apache.org wrote: Salut Jeff, Current Overlay support with tomcat6/7:run is very limited (only use jars from WEB-INF/lib of the the war dependencies). Perso, I use maven-dependency-plugin to extract war content (see sample in this pom [1] ). I agree it's hackhish :-) and having a better support as in the war plugin could be better. But didn't yet have any time to work on that. (can you create an issue for that ?) As my goal was to cut a release soon (ideally starting release process next week, I'm not sure I will have time to work on that) -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy [1] http://svn.apache.org/repos/asf/archiva/trunk/archiva-modules/archiva-web/archiva-webapp/pom.xml 2012/4/28 Jeff MAURY jeffma...@jeffmaury.com: Hello, I am facing the following problem with WAR overlays: I have a first WAR, called skeleton, that contains all necessary stuff: base web.xml, index.jsp and JAR dependencies. I have another WAR whose first dependency is the skeleton which is (as of yet) almost empty except for the slf4j_log4 dependency. When i run tomcat7:run on the skeleton, everything is ok When I run tomcat7:run on the second war, Tomcat start but I am not able to use the application. I have look at the mojo code and it seems it is using the web app source directory (src/main/webapp) by default so as it is empty my case, it cannot work. AM I missing something ? Thanks Jeff -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
Does Tomcat Maven plugin support WAR overlays ?
Hello, I am facing the following problem with WAR overlays: I have a first WAR, called skeleton, that contains all necessary stuff: base web.xml, index.jsp and JAR dependencies. I have another WAR whose first dependency is the skeleton which is (as of yet) almost empty except for the slf4j_log4 dependency. When i run tomcat7:run on the skeleton, everything is ok When I run tomcat7:run on the second war, Tomcat start but I am not able to use the application. I have look at the mojo code and it seems it is using the web app source directory (src/main/webapp) by default so as it is empty my case, it cannot work. AM I missing something ? Thanks Jeff -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury -- Jeff MAURY Legacy code often differs from its suggested alternative by actually working and scaling. - Bjarne Stroustrup http://www.jeffmaury.com http://riadiscuss.jeffmaury.com http://www.twitter.com/jeffmaury
RE: ssl.SessionId Cache keeps growing
As Tomcat is using JSSE, there is a way to control the size of the SSL session's cache. Define a property called javax.net.ssl.sessionCacheSize with a non zero value: it will lower the total number of entries in the cache. By default, it is 0 so no sizing is performed. The defaut value of the ssltimeout is 86400 seconds ie 1 day. It cannot be changed with a property (on Java5). Regards Jeff MAURY Caldarale, Charles R wrote: From: Yuval Perlov [mailto:[EMAIL PROTECTED] Subject: Re: ssl.SessionId Cache keeps growing Been all over those objects thinking I can set a system wide default. There is the JVM -XX:SoftRefLRUPolicyMSPerMB=n option, where n is defined as Number of milliseconds per MB of free space in the heap. This is used to calculate the maximum time to leave a softly reachable object around. If you set it to zero, the SSLSessionId cache should disappear pretty quickly if you force a major GC every now and then. (The default value is 1000). Try an allocation of a gigantic array to force SoftReferences to be discarded, then clear the reference to the array, and force another GC. Ugly, and it probably has really nasty side effects. This is creative but extremely dangerous since I don't know how much real free memory I have. Doesn't matter - the goal is to force an allocation failure thereby causing a major GC that also processes SoftReference objects. A normal System.gc() call won't do that, since there's no allocation failure involved. The second major GC is required to ensure the SoftReference referents are actually released (not sure if that's still needed with the current GC algorithms). Submit an enhancement request, preferably with a patch for new attributes on the Contector elements to control the behavior. How do I do that? http://tomcat.apache.org/bugreport.html I have a feeling they are keeping the SoftReferences even after their default time has elapsed I suspect the default time limit for SSLSession cache, like the default space limit, is infinite. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/ssl.SessionId-Cache-keeps-growing-tp14999824p15839466.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
