Hey André, are you talking about running System.gc()? That should be possible..
Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -----Ursprüngliche Nachricht----- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote: > Just read this email.. :-) > > I figured out we are not using executor connector... > > > Mit freundlichen Grüßen > David Kumar > Softwareentwickler, B. Sc. > Abteilung Infotech - Interaktiv > TELESTAR-DIGITAL GmbH > Am Weiher 14 > D-56766 Ulmen > http://www.telestar.de/ > > > > > -----Ursprüngliche Nachricht----- > Von: David Kumar [mailto:dku...@telestar.de] > Gesendet: Freitag, 18. Januar 2013 09:11 > An: Tomcat Users List > Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open > > here you are with attachment :-) > > > btw: in mod_jk.log I found some > [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] > ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused > connection. No response has been sent to the client (yet) > [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] > ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed. > > > but realy just a few one... > > > Mit freundlichen Grüßen > David Kumar > Softwareentwickler, B. Sc. > Abteilung Infotech - Interaktiv > TELESTAR-DIGITAL GmbH > Am Weiher 14 > D-56766 Ulmen > > http://www.telestar.de/ > > > > > -----Ursprüngliche Nachricht----- > Von: David Kumar > Gesendet: Freitag, 18. Januar 2013 09:08 > An: 'Tomcat Users List' > Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open > > Hey, > > thanks for reply. I got that about the Apache configuration. Since we had our > problem yesterday, again and there was no error at the apache logs I'm > willing to say that is not the main problem and I have to check that, when my > main problem is solved.. :-) > > > > I agree with you about wrong reporting of service. Its just shown up as afs3 > because these service uses 7009 per default. But I'm using 7009 and 9009 for > ajp. > > > So doesn't this mean there is a connection problems between my Apache and the > tomcats? > > You're right, both Webapps doing the same and are configured identically > besides the ports. > > I'm using more than one database, but all of them are used through a database > pool. If there is a bug, I think I should have found some error at my logs > like no free connection or something like that. As there is no such log entry > I'm willing to say that my database connections processing like they should. > > Basically on each tomcat there are running two services. One is a axis2 > project. Our CRM is posting customer data to this webapp. This data will be > persisted into a database. Depending on the information given by our CRM axis > sends a email. > > The second one is basically a cache for our websites. We have a PIM with all > our product data. These app is gathering all the data from PIM and a CMS and > is merging these information together so that the data can be displayed. > All the mentioned data is hold in different "cache objects". Also some > communication with our ERP and some databases are made trough this app. > > The second app is a REST service. Information will be posted as POST or GET > request to it. Most likely the responses are JSON Object. > > When ever one webApp is reloading (automatically or manually) itself, the > result will be posted to the other tomcat/webapp as a serialized object, so > the other on do not need to reload it self. > > I can't say how many SMB files there are, it is depending on some other stuff > so it is dynamic. > > Attached you can find a printed list by lsof. > > There you can see a really strange thing. Yesterday just tomcat2 had the > problem with to many open files. A few days before it was just tomcat1 having > this problem. > > Now let my answer your question: > > 1. That is hard to say, I guess I have to do some more investigation on our > logfiles. > > 2. / 3. Here is my httpd.conf: > <IfModule mpm_worker_module> > ThreadLimit 25 > StartServers 2 > MaxClients 150 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadsPerChild 25 > MaxRequestsPerChild 4000 > </IfModule> > > we are using worker .... > > And here are our tomcat connectors again: > tomcat1: > > <Connector port="7080" protocol="HTTP/1.1" connectionTimeout="20000" > redirectPort="8443"/> > > > <Connector port="7009" protocol="AJP/1.3" redirectPort="8443"/> > > > tomcat2: > <Connector port="9080" protocol="HTTP/1.1" > connectionTimeout="20000" redirectPort="9443"/> > > <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"/> > > > Okay we are not using executor.. I will check that.. > > You probably read my copy-paste error. I did copy some comments out of out > server config --> Sry again. > > 4. we are using.. > 5. via a multipart message sending to the other tomcat. > 6. I don't think so also because of that the connections are kept open on our > ajp ports. > > I know that "CLOSE_WAIT" means, waiting for connections to be closed, but > wondering that it is not closing.. > > > Thanks again > > Mit freundlichen Grüßen > David Kumar > Softwareentwickler, B. Sc. > Abteilung Infotech - Interaktiv > TELESTAR-DIGITAL GmbH > Am Weiher 14 > D-56766 Ulmen > http://www.telestar.de/ > > > > > -----Ursprüngliche Nachricht----- > Von: Christopher Schultz [mailto:ch...@christopherschultz.net] > Gesendet: Donnerstag, 17. Januar 2013 18:38 > An: Tomcat Users List > Betreff: Re: AW: AW: afs3-rmtsys: connections keept open > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > David, > > On 1/17/13 1:49 AM, David Kumar wrote: >> I just checked /var/logs/apache2/error.logs. And found following >> errors: >> >> [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads >> of MaxClients, consider raising the MaxClients setting [Wed Jan 16 >> 15:14:56 2013] [error] server reached MaxClients setting, consider >> raising the MaxClients setting > > So you are maxing-out your connections: you are experiencing enough > load that your configuration cannot handle any more connections: > requests are being queued by the TCP/IP stack and some requests may be > rejected entirely depending upon the queue length of the socket. > > The first question to ask yourself is whether or not your hardware can > take more than you have it configured to accept. For instance, if your > load average, memory usage, and response time are all reasonable, then > you could probably afford to raise your MaxClients setting in httpd. > > Note that the above has almost nothing to do with Tomcat: it only has > to do with Apache httpd. > >> Yesterday my problem occurred about the same time. > > So, the problem is that Tomcat cannot handle your peak load due to a > file handle limitation. IIRC, your current file handle limit for the > Tomcat process is 4096. > >> I'm checking every five minutes how many open files there are: >> >> count open files started: 01-16-2013_15:10: Count: 775 count open >> files started: 01-16-2013_15:15: Count: 1092 > > Okay. lsof will help you determine how many of those are "real" files > versus sockets. Limiting socket usage might be somewhat easier > depending upon what your application actually does. > >> But maybe the afs3 connection causing the Apache error? > > afs3 is a red herring: you are using port 7009 for AJP communication > between httpd and Tomcat and it's being reported as afs3. This has > nothing to do with afs3 unless you know for a fact that your web > application uses that protocol for something. I don't see any evidence > that afs3 is related to your environment in the slightest. I do see > every indication that you are using port 7009 yourself for AJP so > let's assume that's the truth. > > Let's recap what your webapp(s) actually do to see if we can't figure > out where all your file handles are being used. I'll assume that each > Tomcat is configured (reasonably) identically, other than port numbers > and such. I'll also assume that you are running the same webapp using > the same (virtually) identical configuration and that nothing > pathological is happening (like one process totally going crazy and > making thousands of socket connections due to an application bug). > > First, all processes need access to stdin, stdout, stderr: that's 3 > file handles. Plus all shared libraries required to get the process > and JVM started. Plus everything Java needs. Depending on the OS, > that's about 30 or so to begin with. Then, Tomcat uses /dev/random (or > /dev/urandom) plus it needs to load all of its own libraries from JAR > files. There are about 25 of them, and they generally stay open. So, > we're up to about 55 file handles. Don't worry: we won't be counting > these things one-at-a-time for long. Next, Tomcat has two <Connector>s > defined with default connection sizes. At peak load, they will both be > maxed-out at 200 connections each for a total of 402 file handles (1 > bind file handle + 200 file handles for the connections * 2 > connectors). So, we're up to 457. > > Now, onto your web application. You have to count the number of JAR > files that your web application provides: each one of those likely > consumes another file handle that will stay open. Does your webapp use > a database? If so, do you use a connection pool? How big is the > connection pool? Do you have any leaks? If you use a connection pool > and have no leaks, then you can add 'maxActive' file handles to our > running count. If you don't use a connection pool, then you can add > 400 file handles to your count, because any incoming request on either > of those two connectors could result in a database connection. (I > highly recommend using a connection pool if you aren't already). > > Next, you said this: > >> Both of the tomcats are "synchronising" them self. The send some >> serialized objects via http to each other. > > So the webapps make requests to each other? How? Is there a limit to > the number of connections directly from one Tomcat to another? If not, > then you can add another 400 file handles because any incoming > connection could trigger an HTTP connection to the other Tomcat. (What > happens if an incoming client connection causes a connection to the > other Tomcat... will that Tomcat ever call-back to the first one and > set-up a communication storm?). > >> And both of them getting some file from SMB shares. > > How many files? Every file you open consumes a file handle. If you > close the file, you can reduce your fd footprint, but if you keep lots > of files open... > > If you have a dbcp with size=50 and you limit your cross-Tomcat > connections to, say another 50 and your webapp uses 50 JAR files then > you are looking at 600 or so file handles required to run your webapp > under peak load, not including files that must be opened to satisfy a > particular request. > > So the question is: where are all your fds going? Use lsof to > determine what they are being used for. > > Some suggestions: > > 1. Consider the number of connections you actually need to be able to > handle: for both connectors. Maybe you don't need 200 possible > connections for your HTTP connector. > > 2. Make sure your MaxClients in httpd matches make sense with what > you've got in Tomcat's AJP connector: you want to make sure that you > have enough connections available from httpd->Tomcat that you aren't > making users wait. If you're using prefork MPM that means that > MaxClients should be the same as your <Connector>'s maxThreads setting > (or, better yet, use an <Executor>). > > 3. Use an <Executor>. Right now, you might allocate up to 400 threads > to handle connections from both AJP and HTTP. Maybe you don't need > that. You can share request-processing threads by using an <Executor> > and have both connectors share the same pool. > > 4. Use a DBCP. Just in case you aren't. > > 5. Check to see how you are communicating Tomcat-to-Tomcat: you may > have a problem where too many connections are being opened. > > 6. Check to make sure you don't have any resource leaks: JDBC > connections that aren't closed, files not being closed, etc. etc. > Check to make sure you are closing files that don't need to be open > after they are read. > >> But I can't imagine that might be the problem? I'm wondering why >> the tcp connections with state "CLOSE_WAIT" doesn't get closed. > > http://en.wikipedia.org/wiki/Transmission_Control_Protocol > > - -chris > -----BEGIN PGP SIGNATURE----- > Version: GnuPG/MacGPG2 v2.0.17 (Darwin) > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEAREIAAYFAlD4NvsACgkQ9CaO5/Lv0PC4EACfURhDENZPf28HDIazwPqAqri5 > KqYAni9AOSQZVIdsBtQLoEfDcYkpuf7f > =dEDY > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > ------------------------------------------------------------------------ > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org