Felix, I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) *and I saw the below output, some in CLOSE_WAIT and others in ESTABLISHED. If there are 200 open file descriptors 160 are in CLOSE_WAIT state. When the count for CLOSE_WAIT increases I just have to restart tomcat.
java 65189 tomcat8 715u IPv6 237878311 0t0 TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java 65189 tomcat8 716u IPv6 237848923 0t0 TCP localhost:http-alt->localhost:40568 (CLOSE_WAIT) most of the open files are in CLOSE_WAIT state I do not see anything related to database ip. On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher < felix.schumac...@internetallee.de> wrote: > > Am 22.06.20 um 13:22 schrieb Ayub Khan: > > Felix, > > > > I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and from the > output > > I see majority of them are related to sockets as shown below, some of > them > > point to the jar file of tomcat and others to the log file which is > created. > > > > socket:[2084570754] > > socket:[2084579487] > > socket:[2084578478] > > socket:[2084570167] > > Can you try the other command (lsof -p $(cat ...tomcat.pid))? It should > give a bit more details on the used sockets that the proc directory. > > Felix > > > > > On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher < > > felix.schumac...@internetallee.de> wrote: > > > >> Am 22.06.20 um 11:41 schrieb Ayub Khan: > >>> Chris, > >>> > >>> I am using HikariCP for connection pooling. If the database is leaking > >>> connections then I should see connection not available exception. > >>> > >>> How do I find out which file descriptors are leaking ? these are not > >> files > >>> open on disk as there is no explicit disk file I/O in this application. > >>> > >>> I just use the below command to check for open file descriptors: > >>> > >>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l" > >> You could have a look at the name of the files in the pids proc > directory. > >> > >> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ > >> > >> Or you could use the tool lsof to find the open file descriptors. > >> > >> $ lsof -p $(cat /var/run/tomcat8.pid) > >> > >> For both calls you should first change to the uid of the tomcat user or > >> use sudo as in your example. > >> > >> Felix > >> > >>> Thanks and Regards > >>> Ayub > >>> > >>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz < > >>> ch...@christopherschultz.net> wrote: > >>> > >>> Ayub, > >>> > >>> On 6/20/20 11:51, Ayub Khan wrote: > >>>>>> Sorry we are using 8.0.32 version of tomcat. > >>>>>> > >>>>>> below is the configuration: > >>>>>> > >>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built: Jan > >>>>>> 24 2020 16:24:30 UTC Server number: 8.0.32.0 OS Name: > >>>>>> Linux OS Version: 4.4.0-1087-aws Architecture: amd64 JVM > >>>>>> Version: 1.8.0_181-b13 JVM Vendor: Oracle Corporation > >>>>>> > >>>>>> I use the below command to check the file descriptors: > >>>>>> > >>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l" > >>> So you know there is some kind of increase in file-handle use, but you > >>> don't know what types of file handles are increasing, right? > >>> > >>> Can you try to find out which kinds of file handles are increasing? > >>> > >>> I have a sneaking suspicion that it's your database connections and > >>> not actually files open on the disk. > >>> > >>> Are you using a database connection pool? If not, you should really > >>> use one and limit the number of connections to something sane. If you > >>> are using one, are you monitoring it to see how many connections are > >>> actually being used? Are you sure you are using proper resource > >>> management[1]? Even a single code-path that leaks connections can leak > >>> them quickly under load. > >>> > >>>>>> When there an issue related to broken files, this value keeps > >>>>>> increasing, the only way to bring it down is to remove vm instance > >>>>>> from AWS load balancer.> Which version of tomcat should I install > >>>>>> ? > >>> Tomcat 8.0.x hasn't been supported since its last release on 29 June > >>> 2018. That was 8.0.53. Your release is from 8 February 2016 and is > >>> dangerously out of date (unless you are using the Ubuntu-packaged > >>> version, in which case I hope they kept-up with security patches thee > >>> past 4 years). > >>> > >>> -chris > >>> > >>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz < > >>>>>> ch...@christopherschultz.net> wrote: > >>>>>> > >>>>>> Ayub, > >>>>>> > >>>>>> On 6/19/20 16:46, Ayub Khan wrote: > >>>>>>>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS > >>>>>> Which exact version of Tomcat 8.5? If you aren't running the > >>>>>> latest version (8.5.56), please upgrade and re-test. > >>>>>> > >>>>>>>>> If there is slow response from db I see this stack trace and > >>>>>>>>> the open files goes high and the only way to open files go > >>>>>>>>> down is to remove the instance from Amazon load balancer. > >>>>>>>>> > >>>>>>>>> Is there a way to keep the open files low even when Broken > >>>>>>>>> pipe error is thrown ? > >>>>>> What is your evidence that file handles are being left open? > >>>>>> > >>>>>> Which file handles are being left open? > >>>>>> > >>>>>> -chris > >>>>>>> > >> --------------------------------------------------------------------- > >>>>>>> > >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>>>>> > >>>>>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> > >>>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- -------------------------------------------------------------------- Sun Certified Enterprise Architect 1.5 Sun Certified Java Programmer 1.4 Microsoft Certified Systems Engineer 2000 http://in.linkedin.com/pub/ayub-khan/a/811/b81 mobile:+966-502674604 ---------------------------------------------------------------------- It is proved that Hard Work and kowledge will get you close but attitude will get you there. However, it's the Love of God that will put you over the top!!
