Chris,
If some open file is owned by nginx why would it show up if I run the below
command
sudo lsof -p $(cat /var/run/tomcat8.pid)
On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/23/20 19:17, Ayub Khan wrote:
> > Yes we have nginx as reverse proxy, below is the nginx config. We
> > notice this issue only when there is high number of requests,
> > during non peak hours we do not see this issue.> location
> > /myapp/myservice{ #local machine proxy_pass
> > http://localhost:8080; proxy_http_version 1.1;
> >
> > proxy_set_header Connection $connection_upgrade;
> > proxy_set_header Upgrade $http_upgrade;
> > proxy_set_header Host $host; proxy_set_header
> > X-Real-IP $remote_addr; proxy_set_header
> > X-Forwarded-For $proxy_add_x_forwarded_for;
> >
> >
> > proxy_buffers 16 16k; proxy_buffer_size 32k; }
>
> You might want to read about tuning nginx to drop connections after a
> certain period of time, number of requests, etc. Looks like either a
> bug in nginx or a misconfiguration which allows connections to
> stick-around like this. You may have to ask the nginx people. I have
> no experience with nginx myself, while others here may have some
> experience.
>
> > location / { # if using AWS Load balancer, this bit checks for the
> > presence of the https proto flag. if regular http is found, then
> > issue a redirect
> to hit
> > the https endpoint instead if ($http_x_forwarded_proto != 'https')
> > { rewrite ^ https://$host$request_uri? permanent; }
> >
> > proxy_pass http://127.0.0.1:8080; proxy_http_version
> > 1.1;
> >
> > proxy_set_header Connection $connection_upgrade;
> > proxy_set_header Upgrade $http_upgrade;
> > proxy_set_header Host $host; proxy_set_header
> > X-Real-IP $remote_addr; proxy_set_header
> > X-Forwarded-For $proxy_add_x_forwarded_for;
> >
> >
> > proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >
> > *below is the connector*
> >
> > <Connector port="8080"
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > connectionTimeout="2000" maxThreads="50000" URIEncoding="UTF-8"
> > redirectPort="8443" />
>
> 50k threads is a LOT of threads. Do you expect to handle 50k requests
> simultaneously?
>
> > these ports are random, I am not sure who owns the process.
> >
> > localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866
> > is a random port.
> I'm sure you'll find that 55866 is owned by nginx. netstat will tell you
> .
>
> I think you need to look at your nginx configuration. It would also be
> a great time to upgrade to a supported version of Tomcat. I would
> recommend 8.5.56 or 9.0.36.
>
> - -chris
>
> > On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Ayub,
> >
> > On 6/23/20 16:23, Ayub Khan wrote:
> >>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid) *and I
> >>>> saw the below output, some in CLOSE_WAIT and others in
> >>>> ESTABLISHED. If there are 200 open file descriptors 160 are
> >>>> in CLOSE_WAIT state. When the count for CLOSE_WAIT increases
> >>>> I just have to restart tomcat.
> >>>>
> >>>> java 65189 tomcat8 715u IPv6 237878311
> >>>> 0t0 TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java
> >>>> 65189 tomcat8 716u IPv6 237848923 0t0
> >>>> TCP localhost:http-alt->localhost:40568 (CLOSE_WAIT)
> >
> > These are connections from some process into Tomcat listening on
> > port 8080 (that's what localhost:http-alt is). So what process owns
> > the outgoing connection on port 40568 on the same host?
> >
> > Are you using a reverse proxy?
> >
> >>>> most of the open files are in CLOSE_WAIT state I do not see
> >>>> anything related to database ip.
> >
> > Agreed. It looks like you have a reverse proxy who is losing-track
> > of connections, or who is (re)opening connections when it may be
> > unnecessar y.
> >
> > Can you share your <Connector> configuration from server.xml?
> > Remember to remove any secrets.
> >
> > -chris
> >
> >>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> >>>> felix.schumac...@internetallee.de> wrote:
> >>>>
> >>>>>
> >>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>>>>> Felix,
> >>>>>>
> >>>>>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>>> and from the
> >>>>> output
> >>>>>> I see majority of them are related to sockets as shown
> >>>>>> below, some of
> >>>>> them
> >>>>>> point to the jar file of tomcat and others to the log
> >>>>>> file which is
> >>>>> created.
> >>>>>>
> >>>>>> socket:[2084570754] socket:[2084579487]
> >>>>>> socket:[2084578478] socket:[2084570167]
> >>>>>
> >>>>> Can you try the other command (lsof -p $(cat
> >>>>> ...tomcat.pid))? It should give a bit more details on the
> >>>>> used sockets that the proc directory.
> >>>>>
> >>>>> Felix
> >>>>>
> >>>>>>
> >>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> >>>>>> felix.schumac...@internetallee.de> wrote:
> >>>>>>
> >>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>>>>> Chris,
> >>>>>>>>
> >>>>>>>> I am using HikariCP for connection pooling. If the
> >>>>>>>> database is leaking connections then I should see
> >>>>>>>> connection not available exception.
> >>>>>>>>
> >>>>>>>> How do I find out which file descriptors are leaking
> >>>>>>>> ? these are not
> >>>>>>> files
> >>>>>>>> open on disk as there is no explicit disk file I/O in
> >>>>>>>> this application.
> >>>>>>>>
> >>>>>>>> I just use the below command to check for open file
> >>>>>>>> descriptors:
> >>>>>>>>
> >>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
> >>>>>>>> wc -l"
> >>>>>>> You could have a look at the name of the files in the
> >>>>>>> pids proc
> >>>>> directory.
> >>>>>>>
> >>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>>>>
> >>>>>>> Or you could use the tool lsof to find the open file
> >>>>>>> descriptors.
> >>>>>>>
> >>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>>
> >>>>>>> For both calls you should first change to the uid of
> >>>>>>> the tomcat user or use sudo as in your example.
> >>>>>>>
> >>>>>>> Felix
> >>>>>>>
> >>>>>>>> Thanks and Regards Ayub
> >>>>>>>>
> >>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz
> >>>>>>>> < ch...@christopherschultz.net> wrote:
> >>>>>>>>
> >>>>>>>> Ayub,
> >>>>>>>>
> >>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>>>>> Sorry we are using 8.0.32 version of tomcat.
> >>>>>>>>>>>
> >>>>>>>>>>> below is the configuration:
> >>>>>>>>>>>
> >>>>>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu)
> >>>>>>>>>>> Server built: Jan 24 2020 16:24:30 UTC Server
> >>>>>>>>>>> number: 8.0.32.0 OS Name: Linux OS Version:
> >>>>>>>>>>> 4.4.0-1087-aws Architecture: amd64 JVM
> >>>>>>>>>>> Version: 1.8.0_181-b13 JVM Vendor: Oracle
> >>>>>>>>>>> Corporation
> >>>>>>>>>>>
> >>>>>>>>>>> I use the below command to check the file
> >>>>>>>>>>> descriptors:
> >>>>>>>>>>>
> >>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>> So you know there is some kind of increase in
> >>>>>>>> file-handle use, but you don't know what types of
> >>>>>>>> file handles are increasing, right?
> >>>>>>>>
> >>>>>>>> Can you try to find out which kinds of file handles
> >>>>>>>> are increasing?
> >>>>>>>>
> >>>>>>>> I have a sneaking suspicion that it's your database
> >>>>>>>> connections and not actually files open on the disk.
> >>>>>>>>
> >>>>>>>> Are you using a database connection pool? If not,
> >>>>>>>> you should really use one and limit the number of
> >>>>>>>> connections to something sane. If you are using one,
> >>>>>>>> are you monitoring it to see how many connections are
> >>>>>>>> actually being used? Are you sure you are using
> >>>>>>>> proper resource management[1]? Even a single
> >>>>>>>> code-path that leaks connections can leak them
> >>>>>>>> quickly under load.
> >>>>>>>>
> >>>>>>>>>>> When there an issue related to broken files,
> >>>>>>>>>>> this value keeps increasing, the only way to
> >>>>>>>>>>> bring it down is to remove vm instance from AWS
> >>>>>>>>>>> load balancer.> Which version of tomcat should
> >>>>>>>>>>> I install ?
> >>>>>>>> Tomcat 8.0.x hasn't been supported since its last
> >>>>>>>> release on 29 June 2018. That was 8.0.53. Your
> >>>>>>>> release is from 8 February 2016 and is dangerously
> >>>>>>>> out of date (unless you are using the Ubuntu-packaged
> >>>>>>>> version, in which case I hope they kept-up with
> >>>>>>>> security patches thee past 4 years).
> >>>>>>>>
> >>>>>>>> -chris
> >>>>>>>>
> >>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher
> >>>>>>>>>>> Schultz < ch...@christopherschultz.net> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Ayub,
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>>>>>>> tomcat 8.5 broken pipe increases open
> >>>>>>>>>>>>>> files on ubuntu AWS
> >>>>>>>>>>> Which exact version of Tomcat 8.5? If you
> >>>>>>>>>>> aren't running the latest version (8.5.56),
> >>>>>>>>>>> please upgrade and re-test.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> If there is slow response from db I see
> >>>>>>>>>>>>>> this stack trace and the open files goes
> >>>>>>>>>>>>>> high and the only way to open files go
> >>>>>>>>>>>>>> down is to remove the instance from
> >>>>>>>>>>>>>> Amazon load balancer.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is there a way to keep the open files low
> >>>>>>>>>>>>>> even when Broken pipe error is thrown ?
> >>>>>>>>>>> What is your evidence that file handles are
> >>>>>>>>>>> being left open?
> >>>>>>>>>>>
> >>>>>>>>>>> Which file handles are being left open?
> >>>>>>>>>>>
> >>>>>>>>>>> -chris
> >>>>>>>>>>>>
> >>>>>>> ----------------------------------------------------------------
> - ---
> >>
> >>
> >>>>>>>
> - ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For
> >> additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zaP0ACgkQHPApP6U8
> pFiIWw//QaDZsLlbNyYgB2XVjQzMtI0L93TnroEW+MwKnle9UNT0xYBOBk+OnFgB
> nO9j1RLvbE8sROcLKv2KucZYXFxvVfx6eAp3t2OIQkaSRvjnI2Gb/ILPgI7zTWq9
> /1+qjG14A1Gcxy07elRBFik2BO/XqQ6eHiRw0426Ogh7f/djFKVX6qP4TIsFWNee
> shZXLt9ZA/qtmhiNyis+MdpSMenUjmZ5z9TI9iyMQirbannAxl+oVds/0rYSKZ7G
> kabydTSIQpR9OBTJdTrHX79UagkulmsKSKm9f1tXaSW8WwnnEjI3tQIee5orhzHU
> ZdAkMWKU9YvZC8gBLuxs6QHmHEuoxVMRGsTatnfKIdRYIPkDEqMrzpiM7AozFutS
> 8tFMMjQsJYBKM3CqVFr3f7WJVBYjmKMBfH6ItTwZ88RBzOMdz5Odx3JDJMQbsEH5
> ylKQH9Gluqad93he0t+Uaabk6JnCdXbCpdNWyFa/fPnMp4vHFlOb7f62+UlmDT+4
> DyKTl3BtDe9J6pQ0eVY4deUqFgrLCU7C4WVkOqmj9DFmFJxCMp98zXCN2h4O9lhi
> OE853x4UvrPkJMPddN2pg0cJiS6Z3OQzDES1yBVbduga2LnLOJrzDpRh6NtASmNM
> YC82oSScqppOgRGHKWJUrnDTyXjEeJW/eTqhnMIHHXGlwuUfVZ0=
> =zQdW
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
