Chris,
Yes we have nginx as reverse proxy, below is the nginx config. We notice
this issue only when there is high number of requests, during non peak
hours we do not see this issue.
location /myapp/myservice{
#local machine
proxy_pass http://localhost:8080;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffers 16 16k;
proxy_buffer_size 32k;
}
location / {
# if using AWS Load balancer, this bit checks for the presence of the
https proto flag. if regular http is found, then issue a redirect to hit
the https endpoint instead
if ($http_x_forwarded_proto != 'https') {
rewrite ^ https://$host$request_uri? permanent;
}
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffers 16 16k;
proxy_buffer_size 32k;
}
*below is the connector*
<Connector port="8080"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="2000" maxThreads="50000"
URIEncoding="UTF-8"
redirectPort="8443" />
these ports are random, I am not sure who owns the process.
localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866 is a
random port.
On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/23/20 16:23, Ayub Khan wrote:
> > I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) *and I saw
> > the below output, some in CLOSE_WAIT and others in ESTABLISHED. If
> > there are 200 open file descriptors 160 are in CLOSE_WAIT state.
> > When the count for CLOSE_WAIT increases I just have to restart
> > tomcat.
> >
> > java 65189 tomcat8 715u IPv6 237878311 0t0
> > TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java 65189
> > tomcat8 716u IPv6 237848923 0t0 TCP
> > localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>
> These are connections from some process into Tomcat listening on port
> 8080 (that's what localhost:http-alt is). So what process owns the
> outgoing connection on port 40568 on the same host?
>
> Are you using a reverse proxy?
>
> > most of the open files are in CLOSE_WAIT state I do not see
> > anything related to database ip.
>
> Agreed. It looks like you have a reverse proxy who is losing-track of
> connections, or who is (re)opening connections when it may be unnecessar
> y.
>
> Can you share your <Connector> configuration from server.xml? Remember
> to remove any secrets.
>
> - -chris
>
> > On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> > felix.schumac...@internetallee.de> wrote:
> >
> >>
> >> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>> Felix,
> >>>
> >>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and
> >>> from the
> >> output
> >>> I see majority of them are related to sockets as shown below,
> >>> some of
> >> them
> >>> point to the jar file of tomcat and others to the log file
> >>> which is
> >> created.
> >>>
> >>> socket:[2084570754] socket:[2084579487] socket:[2084578478]
> >>> socket:[2084570167]
> >>
> >> Can you try the other command (lsof -p $(cat ...tomcat.pid))? It
> >> should give a bit more details on the used sockets that the proc
> >> directory.
> >>
> >> Felix
> >>
> >>>
> >>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> >>> felix.schumac...@internetallee.de> wrote:
> >>>
> >>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>> Chris,
> >>>>>
> >>>>> I am using HikariCP for connection pooling. If the database
> >>>>> is leaking connections then I should see connection not
> >>>>> available exception.
> >>>>>
> >>>>> How do I find out which file descriptors are leaking ?
> >>>>> these are not
> >>>> files
> >>>>> open on disk as there is no explicit disk file I/O in this
> >>>>> application.
> >>>>>
> >>>>> I just use the below command to check for open file
> >>>>> descriptors:
> >>>>>
> >>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc
> >>>>> -l"
> >>>> You could have a look at the name of the files in the pids
> >>>> proc
> >> directory.
> >>>>
> >>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>
> >>>> Or you could use the tool lsof to find the open file
> >>>> descriptors.
> >>>>
> >>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>
> >>>> For both calls you should first change to the uid of the
> >>>> tomcat user or use sudo as in your example.
> >>>>
> >>>> Felix
> >>>>
> >>>>> Thanks and Regards Ayub
> >>>>>
> >>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> >>>>> ch...@christopherschultz.net> wrote:
> >>>>>
> >>>>> Ayub,
> >>>>>
> >>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>> Sorry we are using 8.0.32 version of tomcat.
> >>>>>>>>
> >>>>>>>> below is the configuration:
> >>>>>>>>
> >>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server
> >>>>>>>> built: Jan 24 2020 16:24:30 UTC Server number:
> >>>>>>>> 8.0.32.0 OS Name: Linux OS Version:
> >>>>>>>> 4.4.0-1087-aws Architecture: amd64 JVM Version:
> >>>>>>>> 1.8.0_181-b13 JVM Vendor: Oracle Corporation
> >>>>>>>>
> >>>>>>>> I use the below command to check the file
> >>>>>>>> descriptors:
> >>>>>>>>
> >>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
> >>>>>>>> wc -l"
> >>>>> So you know there is some kind of increase in file-handle
> >>>>> use, but you don't know what types of file handles are
> >>>>> increasing, right?
> >>>>>
> >>>>> Can you try to find out which kinds of file handles are
> >>>>> increasing?
> >>>>>
> >>>>> I have a sneaking suspicion that it's your database
> >>>>> connections and not actually files open on the disk.
> >>>>>
> >>>>> Are you using a database connection pool? If not, you
> >>>>> should really use one and limit the number of connections
> >>>>> to something sane. If you are using one, are you monitoring
> >>>>> it to see how many connections are actually being used? Are
> >>>>> you sure you are using proper resource management[1]? Even
> >>>>> a single code-path that leaks connections can leak them
> >>>>> quickly under load.
> >>>>>
> >>>>>>>> When there an issue related to broken files, this
> >>>>>>>> value keeps increasing, the only way to bring it down
> >>>>>>>> is to remove vm instance from AWS load balancer.>
> >>>>>>>> Which version of tomcat should I install ?
> >>>>> Tomcat 8.0.x hasn't been supported since its last release
> >>>>> on 29 June 2018. That was 8.0.53. Your release is from 8
> >>>>> February 2016 and is dangerously out of date (unless you
> >>>>> are using the Ubuntu-packaged version, in which case I hope
> >>>>> they kept-up with security patches thee past 4 years).
> >>>>>
> >>>>> -chris
> >>>>>
> >>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz
> >>>>>>>> < ch...@christopherschultz.net> wrote:
> >>>>>>>>
> >>>>>>>> Ayub,
> >>>>>>>>
> >>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>>>> tomcat 8.5 broken pipe increases open files on
> >>>>>>>>>>> ubuntu AWS
> >>>>>>>> Which exact version of Tomcat 8.5? If you aren't
> >>>>>>>> running the latest version (8.5.56), please upgrade
> >>>>>>>> and re-test.
> >>>>>>>>
> >>>>>>>>>>> If there is slow response from db I see this
> >>>>>>>>>>> stack trace and the open files goes high and
> >>>>>>>>>>> the only way to open files go down is to remove
> >>>>>>>>>>> the instance from Amazon load balancer.
> >>>>>>>>>>>
> >>>>>>>>>>> Is there a way to keep the open files low even
> >>>>>>>>>>> when Broken pipe error is thrown ?
> >>>>>>>> What is your evidence that file handles are being
> >>>>>>>> left open?
> >>>>>>>>
> >>>>>>>> Which file handles are being left open?
> >>>>>>>>
> >>>>>>>> -chris
> >>>>>>>>>
> >>>> -------------------------------------------------------------------
> - --
> >>>>>>>>>
> >>>>>
> >>>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>>>>>>> For additional commands, e-mail:
> >>>>>>>>> users-h...@tomcat.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> -----------------------------------------------------------------
> - ----
> >>>>>>
> >>>>>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>>>> For additional commands, e-mail:
> >>>>>> users-h...@tomcat.apache.org
> >>>>>>
> >>>>>>
> >>>>
> >>>> -------------------------------------------------------------------
> - --
> >>>>
> >>>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>> For additional commands, e-mail:
> >>>> users-h...@tomcat.apache.org
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7yeJYACgkQHPApP6U8
> pFhaHQ/9E56+1vZwl0kb3p/74AwmPkQ0wRJB08CYNpAxLHCDaQ4PJrMqGQDBfFcs
> RT/ithuxS8qJXyd02YgF6hWs8OtiAkODsPqjNxhuQMrLuF7qt2x0CgPWSfMKZVLW
> gLNA09kI6pQzfQrQROhfcqBQqGzaHVN8W4m+wTYOyCbMi2IjBF6lxcMtlCw7org/
> AFe904+ST8D2O+S8CK97dP+74oIu1gEJniLxs6kypx9+qzpgesmhhd6H/965XCBp
> ZEK6AwqedGUUPXxqxRdp6YmqflddtZdL4W1GhqbKN055+xGL+6rIlHD9W//ERJE8
> JutSaX6K5Z1t8+6K48Fwkju549g63J099fKQftg5hsSZ8Xwuqo9B3qOn2HcqrSZA
> +pgkdDDY5m0cHnB71GEREikABU36YnhBDeI0O4ocoQlRNnnIMAfb4hFxdwRCp5Qs
> dRM/DyFwRvdbiOQ0hfoUhx9HDH6JRsWrs5m2hy4n9m+5DJcQ/fxjUcqq0LTzIP3v
> upu81THiYVKBgbKpLKxadQnAzCj+TshEVF/E5D7ZWKG+enJAmnfrnIj5kKG/65HV
> RozUoE69+p1xWN6mWtZq8hMVjKfX1p4z+XlgTg03K/VcdNvs5gE4aKrevx1Ueueb
> hjogi6+JlQA7ysDlxHsttXrGVaY4DV9/c3ig8g7/pDQJCuoFe78=
> =BMkY
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
