Hey,
we are still having that issue.
But we could manage to figure out some more stuff.
We made a Tomcat and Java update since that time we had our problem a few times
again, also we did some reconfiguration with connectors etc.
The last 2 times we where able to see, that both tomcat by them self where
alive. Just ajp on both where dead. We couldn't make a connection either trough
7009 nor 9009. An with our openFiles trick we found a lot of close_wait again,
e.g. 200 for 9009. I left the second tomcat on this state for a few ours just
to see, what happens. The count of 200 connection with close_wait was kept
until a reboot of the tomcat.
I would say with some of our reconfiguration we managed to stop increasing
connections. But still we are not sure why our ajp connections dying..
Here is our connector out of Server.xml:
<Connector port="9009" protocol="AJP/1.3" redirectPort="9443"
maxThreads="200" connectionTimeout="600000" />
worker.properties:
worker.tomcatX.host=localhost
worker.tomcatX.type=ajp13
worker.tomcatX.fail_on_status=404
worker.tomcatX.lbfactor=1
worker.tomcatX.ping_timeout=1000
worker.tomcatX.ping_mode=A
worker.tomcatX.socket_timeout=10
worker.tomcatX.connection_pool_timeout=600
worker.tomcat1.reference=worker.tomcatX
worker.tomcat1.port=7009
worker.tomcat2.reference=worker.tomcatX
worker.tomcat2.port=9009
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.status.type=status
Hopefully one of you guys can give us a hint to fix that problem.
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Dienstag, 22. Januar 2013 07:36
An: Tomcat Users List
Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
Hey,
last friday I changed our configuration to use a executor.
Here is what I did:
<Connector port="7009" protocol="AJP/1.3" redirectPort="8443" maxThreads="200"
executor="active-executor" />
<Executor name="active-executor" namePrefix="activeThread-" maxThreads="200"
minSpareThreads="30" maxIdleTime="60000" />
<Connector port="7080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" executor="active-executor" />
The second tomcat has same configuration besides ports..
Until yesterday it worked like a charm. But at late afternoon one of the
tomcats failed again..
I couldn't start the garbagecollection so far..
Any other ideas?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Freitag, 18. Januar 2013 11:19
An: Tomcat Users List; Tomcat Users List
Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept
open
Hey,
I do that at next deployment. --> I Thursday...
So far I'm trying executor for tomcat. As far I read, when I'm using connectors
idle process are forced to be close..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Freitag, 18. Januar 2013 11:10
An: Tomcat Users List
Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept
open
David Kumar wrote:
> Hey André,
>
> are you talking about running System.gc()?
Yes.
> That should be possible..
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: André Warnier [mailto:a...@ice-sa.com]
> Gesendet: Freitag, 18. Januar 2013 10:07
> An: Tomcat Users List
> Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept
> open
>
> David,
> (and sorry for top-posting here)
>
> just to verify something.
> Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a
> moment when you
> have all these connections in CLOSE_WAIT, and see if they disappear after the
> GC ?
>
> If yes, it may give a good clue about where all these CLOSE_WAITs are coming
> from.
>
>
> David Kumar wrote:
>> Just read this email.. :-)
>>
>> I figured out we are not using executor connector...
>>
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: David Kumar [mailto:dku...@telestar.de]
>> Gesendet: Freitag, 18. Januar 2013 09:11
>> An: Tomcat Users List
>> Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> here you are with attachment :-)
>>
>>
>> btw: in mod_jk.log I found some
>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error]
>> ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused
>> connection. No response has been sent to the client (yet)
>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error]
>> ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed.
>>
>>
>> but realy just a few one...
>>
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>>
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: David Kumar
>> Gesendet: Freitag, 18. Januar 2013 09:08
>> An: 'Tomcat Users List'
>> Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> Hey,
>>
>> thanks for reply. I got that about the Apache configuration. Since we had
>> our problem yesterday, again and there was no error at the apache logs I'm
>> willing to say that is not the main problem and I have to check that, when
>> my main problem is solved.. :-)
>>
>>
>>
>> I agree with you about wrong reporting of service. Its just shown up as afs3
>> because these service uses 7009 per default. But I'm using 7009 and 9009 for
>> ajp.
>>
>>
>> So doesn't this mean there is a connection problems between my Apache and
>> the tomcats?
>>
>> You're right, both Webapps doing the same and are configured identically
>> besides the ports.
>>
>> I'm using more than one database, but all of them are used through a
>> database pool. If there is a bug, I think I should have found some error at
>> my logs like no free connection or something like that. As there is no such
>> log entry I'm willing to say that my database connections processing like
>> they should.
>>
>> Basically on each tomcat there are running two services. One is a axis2
>> project. Our CRM is posting customer data to this webapp. This data will be
>> persisted into a database. Depending on the information given by our CRM
>> axis sends a email.
>>
>> The second one is basically a cache for our websites. We have a PIM with all
>> our product data. These app is gathering all the data from PIM and a CMS and
>> is merging these information together so that the data can be displayed.
>> All the mentioned data is hold in different "cache objects". Also some
>> communication with our ERP and some databases are made trough this app.
>>
>> The second app is a REST service. Information will be posted as POST or GET
>> request to it. Most likely the responses are JSON Object.
>>
>> When ever one webApp is reloading (automatically or manually) itself, the
>> result will be posted to the other tomcat/webapp as a serialized object, so
>> the other on do not need to reload it self.
>>
>> I can't say how many SMB files there are, it is depending on some other
>> stuff so it is dynamic.
>>
>> Attached you can find a printed list by lsof.
>>
>> There you can see a really strange thing. Yesterday just tomcat2 had the
>> problem with to many open files. A few days before it was just tomcat1
>> having this problem.
>>
>> Now let my answer your question:
>>
>> 1. That is hard to say, I guess I have to do some more investigation on our
>> logfiles.
>>
>> 2. / 3. Here is my httpd.conf:
>> <IfModule mpm_worker_module>
>> ThreadLimit 25
>> StartServers 2
>> MaxClients 150
>> MinSpareThreads 25
>> MaxSpareThreads 75
>> ThreadsPerChild 25
>> MaxRequestsPerChild 4000
>> </IfModule>
>>
>> we are using worker ....
>>
>> And here are our tomcat connectors again:
>> tomcat1:
>>
>> <Connector port="7080" protocol="HTTP/1.1" connectionTimeout="20000"
>> redirectPort="8443"/>
>>
>>
>> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443"/>
>>
>>
>> tomcat2:
>> <Connector port="9080" protocol="HTTP/1.1"
>> connectionTimeout="20000" redirectPort="9443"/>
>>
>> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"/>
>>
>>
>> Okay we are not using executor.. I will check that..
>>
>> You probably read my copy-paste error. I did copy some comments out of out
>> server config --> Sry again.
>>
>> 4. we are using..
>> 5. via a multipart message sending to the other tomcat.
>> 6. I don't think so also because of that the connections are kept open on
>> our ajp ports.
>>
>> I know that "CLOSE_WAIT" means, waiting for connections to be closed, but
>> wondering that it is not closing..
>>
>>
>> Thanks again
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Christopher Schultz [mailto:ch...@christopherschultz.net]
>> Gesendet: Donnerstag, 17. Januar 2013 18:38
>> An: Tomcat Users List
>> Betreff: Re: AW: AW: afs3-rmtsys: connections keept open
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> David,
>>
>> On 1/17/13 1:49 AM, David Kumar wrote:
>>> I just checked /var/logs/apache2/error.logs. And found following
>>> errors:
>>>
>>> [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads
>>> of MaxClients, consider raising the MaxClients setting [Wed Jan 16
>>> 15:14:56 2013] [error] server reached MaxClients setting, consider
>>> raising the MaxClients setting
>> So you are maxing-out your connections: you are experiencing enough
>> load that your configuration cannot handle any more connections:
>> requests are being queued by the TCP/IP stack and some requests may be
>> rejected entirely depending upon the queue length of the socket.
>>
>> The first question to ask yourself is whether or not your hardware can
>> take more than you have it configured to accept. For instance, if your
>> load average, memory usage, and response time are all reasonable, then
>> you could probably afford to raise your MaxClients setting in httpd.
>>
>> Note that the above has almost nothing to do with Tomcat: it only has
>> to do with Apache httpd.
>>
>>> Yesterday my problem occurred about the same time.
>> So, the problem is that Tomcat cannot handle your peak load due to a
>> file handle limitation. IIRC, your current file handle limit for the
>> Tomcat process is 4096.
>>
>>> I'm checking every five minutes how many open files there are:
>>>
>>> count open files started: 01-16-2013_15:10: Count: 775 count open
>>> files started: 01-16-2013_15:15: Count: 1092
>> Okay. lsof will help you determine how many of those are "real" files
>> versus sockets. Limiting socket usage might be somewhat easier
>> depending upon what your application actually does.
>>
>>> But maybe the afs3 connection causing the Apache error?
>> afs3 is a red herring: you are using port 7009 for AJP communication
>> between httpd and Tomcat and it's being reported as afs3. This has
>> nothing to do with afs3 unless you know for a fact that your web
>> application uses that protocol for something. I don't see any evidence
>> that afs3 is related to your environment in the slightest. I do see
>> every indication that you are using port 7009 yourself for AJP so
>> let's assume that's the truth.
>>
>> Let's recap what your webapp(s) actually do to see if we can't figure
>> out where all your file handles are being used. I'll assume that each
>> Tomcat is configured (reasonably) identically, other than port numbers
>> and such. I'll also assume that you are running the same webapp using
>> the same (virtually) identical configuration and that nothing
>> pathological is happening (like one process totally going crazy and
>> making thousands of socket connections due to an application bug).
>>
>> First, all processes need access to stdin, stdout, stderr: that's 3
>> file handles. Plus all shared libraries required to get the process
>> and JVM started. Plus everything Java needs. Depending on the OS,
>> that's about 30 or so to begin with. Then, Tomcat uses /dev/random (or
>> /dev/urandom) plus it needs to load all of its own libraries from JAR
>> files. There are about 25 of them, and they generally stay open. So,
>> we're up to about 55 file handles. Don't worry: we won't be counting
>> these things one-at-a-time for long. Next, Tomcat has two <Connector>s
>> defined with default connection sizes. At peak load, they will both be
>> maxed-out at 200 connections each for a total of 402 file handles (1
>> bind file handle + 200 file handles for the connections * 2
>> connectors). So, we're up to 457.
>>
>> Now, onto your web application. You have to count the number of JAR
>> files that your web application provides: each one of those likely
>> consumes another file handle that will stay open. Does your webapp use
>> a database? If so, do you use a connection pool? How big is the
>> connection pool? Do you have any leaks? If you use a connection pool
>> and have no leaks, then you can add 'maxActive' file handles to our
>> running count. If you don't use a connection pool, then you can add
>> 400 file handles to your count, because any incoming request on either
>> of those two connectors could result in a database connection. (I
>> highly recommend using a connection pool if you aren't already).
>>
>> Next, you said this:
>>
>>> Both of the tomcats are "synchronising" them self. The send some
>>> serialized objects via http to each other.
>> So the webapps make requests to each other? How? Is there a limit to
>> the number of connections directly from one Tomcat to another? If not,
>> then you can add another 400 file handles because any incoming
>> connection could trigger an HTTP connection to the other Tomcat. (What
>> happens if an incoming client connection causes a connection to the
>> other Tomcat... will that Tomcat ever call-back to the first one and
>> set-up a communication storm?).
>>
>>> And both of them getting some file from SMB shares.
>> How many files? Every file you open consumes a file handle. If you
>> close the file, you can reduce your fd footprint, but if you keep lots
>> of files open...
>>
>> If you have a dbcp with size=50 and you limit your cross-Tomcat
>> connections to, say another 50 and your webapp uses 50 JAR files then
>> you are looking at 600 or so file handles required to run your webapp
>> under peak load, not including files that must be opened to satisfy a
>> particular request.
>>
>> So the question is: where are all your fds going? Use lsof to
>> determine what they are being used for.
>>
>> Some suggestions:
>>
>> 1. Consider the number of connections you actually need to be able to
>> handle: for both connectors. Maybe you don't need 200 possible
>> connections for your HTTP connector.
>>
>> 2. Make sure your MaxClients in httpd matches make sense with what
>> you've got in Tomcat's AJP connector: you want to make sure that you
>> have enough connections available from httpd->Tomcat that you aren't
>> making users wait. If you're using prefork MPM that means that
>> MaxClients should be the same as your <Connector>'s maxThreads setting
>> (or, better yet, use an <Executor>).
>>
>> 3. Use an <Executor>. Right now, you might allocate up to 400 threads
>> to handle connections from both AJP and HTTP. Maybe you don't need
>> that. You can share request-processing threads by using an <Executor>
>> and have both connectors share the same pool.
>>
>> 4. Use a DBCP. Just in case you aren't.
>>
>> 5. Check to see how you are communicating Tomcat-to-Tomcat: you may
>> have a problem where too many connections are being opened.
>>
>> 6. Check to make sure you don't have any resource leaks: JDBC
>> connections that aren't closed, files not being closed, etc. etc.
>> Check to make sure you are closing files that don't need to be open
>> after they are read.
>>
>>> But I can't imagine that might be the problem? I'm wondering why
>>> the tcp connections with state "CLOSE_WAIT" doesn't get closed.
>> http://en.wikipedia.org/wiki/Transmission_Control_Protocol
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iEYEAREIAAYFAlD4NvsACgkQ9CaO5/Lv0PC4EACfURhDENZPf28HDIazwPqAqri5
>> KqYAni9AOSQZVIdsBtQLoEfDcYkpuf7f
>> =dEDY
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
