-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Ayub,
On 6/23/20 16:23, Ayub Khan wrote: > I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) *and I saw > the below output, some in CLOSE_WAIT and others in ESTABLISHED. If > there are 200 open file descriptors 160 are in CLOSE_WAIT state. > When the count for CLOSE_WAIT increases I just have to restart > tomcat. > > java 65189 tomcat8 715u IPv6 237878311 0t0 > TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java 65189 > tomcat8 716u IPv6 237848923 0t0 TCP > localhost:http-alt->localhost:40568 (CLOSE_WAIT) These are connections from some process into Tomcat listening on port 8080 (that's what localhost:http-alt is). So what process owns the outgoing connection on port 40568 on the same host? Are you using a reverse proxy? > most of the open files are in CLOSE_WAIT state I do not see > anything related to database ip. Agreed. It looks like you have a reverse proxy who is losing-track of connections, or who is (re)opening connections when it may be unnecessar y. Can you share your <Connector> configuration from server.xml? Remember to remove any secrets. - -chris > On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher < > felix.schumac...@internetallee.de> wrote: > >> >> Am 22.06.20 um 13:22 schrieb Ayub Khan: >>> Felix, >>> >>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and >>> from the >> output >>> I see majority of them are related to sockets as shown below, >>> some of >> them >>> point to the jar file of tomcat and others to the log file >>> which is >> created. >>> >>> socket:[2084570754] socket:[2084579487] socket:[2084578478] >>> socket:[2084570167] >> >> Can you try the other command (lsof -p $(cat ...tomcat.pid))? It >> should give a bit more details on the used sockets that the proc >> directory. >> >> Felix >> >>> >>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher < >>> felix.schumac...@internetallee.de> wrote: >>> >>>> Am 22.06.20 um 11:41 schrieb Ayub Khan: >>>>> Chris, >>>>> >>>>> I am using HikariCP for connection pooling. If the database >>>>> is leaking connections then I should see connection not >>>>> available exception. >>>>> >>>>> How do I find out which file descriptors are leaking ? >>>>> these are not >>>> files >>>>> open on disk as there is no explicit disk file I/O in this >>>>> application. >>>>> >>>>> I just use the below command to check for open file >>>>> descriptors: >>>>> >>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc >>>>> -l" >>>> You could have a look at the name of the files in the pids >>>> proc >> directory. >>>> >>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ >>>> >>>> Or you could use the tool lsof to find the open file >>>> descriptors. >>>> >>>> $ lsof -p $(cat /var/run/tomcat8.pid) >>>> >>>> For both calls you should first change to the uid of the >>>> tomcat user or use sudo as in your example. >>>> >>>> Felix >>>> >>>>> Thanks and Regards Ayub >>>>> >>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz < >>>>> ch...@christopherschultz.net> wrote: >>>>> >>>>> Ayub, >>>>> >>>>> On 6/20/20 11:51, Ayub Khan wrote: >>>>>>>> Sorry we are using 8.0.32 version of tomcat. >>>>>>>> >>>>>>>> below is the configuration: >>>>>>>> >>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server >>>>>>>> built: Jan 24 2020 16:24:30 UTC Server number: >>>>>>>> 8.0.32.0 OS Name: Linux OS Version: >>>>>>>> 4.4.0-1087-aws Architecture: amd64 JVM Version: >>>>>>>> 1.8.0_181-b13 JVM Vendor: Oracle Corporation >>>>>>>> >>>>>>>> I use the below command to check the file >>>>>>>> descriptors: >>>>>>>> >>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | >>>>>>>> wc -l" >>>>> So you know there is some kind of increase in file-handle >>>>> use, but you don't know what types of file handles are >>>>> increasing, right? >>>>> >>>>> Can you try to find out which kinds of file handles are >>>>> increasing? >>>>> >>>>> I have a sneaking suspicion that it's your database >>>>> connections and not actually files open on the disk. >>>>> >>>>> Are you using a database connection pool? If not, you >>>>> should really use one and limit the number of connections >>>>> to something sane. If you are using one, are you monitoring >>>>> it to see how many connections are actually being used? Are >>>>> you sure you are using proper resource management[1]? Even >>>>> a single code-path that leaks connections can leak them >>>>> quickly under load. >>>>> >>>>>>>> When there an issue related to broken files, this >>>>>>>> value keeps increasing, the only way to bring it down >>>>>>>> is to remove vm instance from AWS load balancer.> >>>>>>>> Which version of tomcat should I install ? >>>>> Tomcat 8.0.x hasn't been supported since its last release >>>>> on 29 June 2018. That was 8.0.53. Your release is from 8 >>>>> February 2016 and is dangerously out of date (unless you >>>>> are using the Ubuntu-packaged version, in which case I hope >>>>> they kept-up with security patches thee past 4 years). >>>>> >>>>> -chris >>>>> >>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz >>>>>>>> < ch...@christopherschultz.net> wrote: >>>>>>>> >>>>>>>> Ayub, >>>>>>>> >>>>>>>> On 6/19/20 16:46, Ayub Khan wrote: >>>>>>>>>>> tomcat 8.5 broken pipe increases open files on >>>>>>>>>>> ubuntu AWS >>>>>>>> Which exact version of Tomcat 8.5? If you aren't >>>>>>>> running the latest version (8.5.56), please upgrade >>>>>>>> and re-test. >>>>>>>> >>>>>>>>>>> If there is slow response from db I see this >>>>>>>>>>> stack trace and the open files goes high and >>>>>>>>>>> the only way to open files go down is to remove >>>>>>>>>>> the instance from Amazon load balancer. >>>>>>>>>>> >>>>>>>>>>> Is there a way to keep the open files low even >>>>>>>>>>> when Broken pipe error is thrown ? >>>>>>>> What is your evidence that file handles are being >>>>>>>> left open? >>>>>>>> >>>>>>>> Which file handles are being left open? >>>>>>>> >>>>>>>> -chris >>>>>>>>> >>>> ------------------------------------------------------------------- - -- >>>>>>>>> >>>>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>>>> For additional commands, e-mail: >>>>>>>>> users-h...@tomcat.apache.org >>>>>>>>> >>>>>>>>> >>>>>> ----------------------------------------------------------------- - ---- >>>>>> >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: >>>>>> users-h...@tomcat.apache.org >>>>>> >>>>>> >>>> >>>> ------------------------------------------------------------------- - -- >>>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: >>>> users-h...@tomcat.apache.org >>>> >>>> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7yeJYACgkQHPApP6U8 pFhaHQ/9E56+1vZwl0kb3p/74AwmPkQ0wRJB08CYNpAxLHCDaQ4PJrMqGQDBfFcs RT/ithuxS8qJXyd02YgF6hWs8OtiAkODsPqjNxhuQMrLuF7qt2x0CgPWSfMKZVLW gLNA09kI6pQzfQrQROhfcqBQqGzaHVN8W4m+wTYOyCbMi2IjBF6lxcMtlCw7org/ AFe904+ST8D2O+S8CK97dP+74oIu1gEJniLxs6kypx9+qzpgesmhhd6H/965XCBp ZEK6AwqedGUUPXxqxRdp6YmqflddtZdL4W1GhqbKN055+xGL+6rIlHD9W//ERJE8 JutSaX6K5Z1t8+6K48Fwkju549g63J099fKQftg5hsSZ8Xwuqo9B3qOn2HcqrSZA +pgkdDDY5m0cHnB71GEREikABU36YnhBDeI0O4ocoQlRNnnIMAfb4hFxdwRCp5Qs dRM/DyFwRvdbiOQ0hfoUhx9HDH6JRsWrs5m2hy4n9m+5DJcQ/fxjUcqq0LTzIP3v upu81THiYVKBgbKpLKxadQnAzCj+TshEVF/E5D7ZWKG+enJAmnfrnIj5kKG/65HV RozUoE69+p1xWN6mWtZq8hMVjKfX1p4z+XlgTg03K/VcdNvs5gE4aKrevx1Ueueb hjogi6+JlQA7ysDlxHsttXrGVaY4DV9/c3ig8g7/pDQJCuoFe78= =BMkY -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
