Chris,
Ok, I will investigate nginx side as well. Thank you for the pointers
On Wed, 24 Jun 2020, 19:45 Christopher Schultz, <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
>
> Ayub,
>
> On 6/24/20 11:05, Ayub Khan wrote:
> > If some open file is owned by nginx why would it show up if I run
> > the below command> sudo lsof -p $(cat /var/run/tomcat8.pid)
>
> Because network connections have two ends.
>
> - -chris
>
> > On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > Ayub,
> >
> > On 6/23/20 19:17, Ayub Khan wrote:
> >>>> Yes we have nginx as reverse proxy, below is the nginx
> >>>> config. We notice this issue only when there is high number
> >>>> of requests, during non peak hours we do not see this issue.>
> >>>> location /myapp/myservice{ #local machine proxy_pass
> >>>> http://localhost:8080; proxy_http_version 1.1;
> >>>>
> >>>> proxy_set_header Connection $connection_upgrade;
> >>>> proxy_set_header Upgrade $http_upgrade;
> >>>> proxy_set_header Host $host;
> >>>> proxy_set_header X-Real-IP $remote_addr;
> >>>> proxy_set_header X-Forwarded-For
> >>>> $proxy_add_x_forwarded_for;
> >>>>
> >>>>
> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >
> > You might want to read about tuning nginx to drop connections after
> > a certain period of time, number of requests, etc. Looks like
> > either a bug in nginx or a misconfiguration which allows
> > connections to stick-around like this. You may have to ask the
> > nginx people. I have no experience with nginx myself, while others
> > here may have some experience.
> >
> >>>> location / { # if using AWS Load balancer, this bit checks
> >>>> for the presence of the https proto flag. if regular http is
> >>>> found, then issue a redirect
> > to hit
> >>>> the https endpoint instead if ($http_x_forwarded_proto !=
> >>>> 'https') { rewrite ^ https://$host$request_uri? permanent; }
> >>>>
> >>>> proxy_pass http://127.0.0.1:8080;
> >>>> proxy_http_version 1.1;
> >>>>
> >>>> proxy_set_header Connection $connection_upgrade;
> >>>> proxy_set_header Upgrade $http_upgrade;
> >>>> proxy_set_header Host $host;
> >>>> proxy_set_header X-Real-IP $remote_addr;
> >>>> proxy_set_header X-Forwarded-For
> >>>> $proxy_add_x_forwarded_for;
> >>>>
> >>>>
> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >>>>
> >>>> *below is the connector*
> >>>>
> >>>> <Connector port="8080"
> >>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
> >>>> connectionTimeout="2000" maxThreads="50000"
> >>>> URIEncoding="UTF-8" redirectPort="8443" />
> >
> > 50k threads is a LOT of threads. Do you expect to handle 50k
> > requests simultaneously?
> >
> >>>> these ports are random, I am not sure who owns the process.
> >>>>
> >>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port
> >>>> 55866 is a random port.
> > I'm sure you'll find that 55866 is owned by nginx. netstat will
> > tell you .
> >
> > I think you need to look at your nginx configuration. It would also
> > be a great time to upgrade to a supported version of Tomcat. I
> > would recommend 8.5.56 or 9.0.36.
> >
> > -chris
> >
> >>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
> >>>> ch...@christopherschultz.net> wrote:
> >>>>
> >>>> Ayub,
> >>>>
> >>>> On 6/23/20 16:23, Ayub Khan wrote:
> >>>>>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>> *and I saw the below output, some in CLOSE_WAIT and
> >>>>>>> others in ESTABLISHED. If there are 200 open file
> >>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count
> >>>>>>> for CLOSE_WAIT increases I just have to restart
> >>>>>>> tomcat.
> >>>>>>>
> >>>>>>> java 65189 tomcat8 715u IPv6
> >>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760
> >>>>>>> (CLOSE_WAIT) java 65189 tomcat8 716u IPv6
> >>>>>>> 237848923 0t0 TCP
> >>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT)
> >>>>
> >>>> These are connections from some process into Tomcat listening
> >>>> on port 8080 (that's what localhost:http-alt is). So what
> >>>> process owns the outgoing connection on port 40568 on the
> >>>> same host?
> >>>>
> >>>> Are you using a reverse proxy?
> >>>>
> >>>>>>> most of the open files are in CLOSE_WAIT state I do not
> >>>>>>> see anything related to database ip.
> >>>>
> >>>> Agreed. It looks like you have a reverse proxy who is
> >>>> losing-track of connections, or who is (re)opening
> >>>> connections when it may be unnecessar y.
> >>>>
> >>>> Can you share your <Connector> configuration from
> >>>> server.xml? Remember to remove any secrets.
> >>>>
> >>>> -chris
> >>>>
> >>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> >>>>>>> felix.schumac...@internetallee.de> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>>>>>>>> Felix,
> >>>>>>>>>
> >>>>>>>>> I executed ls -l /proc/$(cat
> >>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
> >>>>>>>> output
> >>>>>>>>> I see majority of them are related to sockets as
> >>>>>>>>> shown below, some of
> >>>>>>>> them
> >>>>>>>>> point to the jar file of tomcat and others to the
> >>>>>>>>> log file which is
> >>>>>>>> created.
> >>>>>>>>>
> >>>>>>>>> socket:[2084570754] socket:[2084579487]
> >>>>>>>>> socket:[2084578478] socket:[2084570167]
> >>>>>>>>
> >>>>>>>> Can you try the other command (lsof -p $(cat
> >>>>>>>> ...tomcat.pid))? It should give a bit more details on
> >>>>>>>> the used sockets that the proc directory.
> >>>>>>>>
> >>>>>>>> Felix
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> >>>>>>>>> felix.schumac...@internetallee.de> wrote:
> >>>>>>>>>
> >>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>>>>>>>> Chris,
> >>>>>>>>>>>
> >>>>>>>>>>> I am using HikariCP for connection pooling. If
> >>>>>>>>>>> the database is leaking connections then I
> >>>>>>>>>>> should see connection not available exception.
> >>>>>>>>>>>
> >>>>>>>>>>> How do I find out which file descriptors are
> >>>>>>>>>>> leaking ? these are not
> >>>>>>>>>> files
> >>>>>>>>>>> open on disk as there is no explicit disk file
> >>>>>>>>>>> I/O in this application.
> >>>>>>>>>>>
> >>>>>>>>>>> I just use the below command to check for open
> >>>>>>>>>>> file descriptors:
> >>>>>>>>>>>
> >>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>>>> You could have a look at the name of the files in
> >>>>>>>>>> the pids proc
> >>>>>>>> directory.
> >>>>>>>>>>
> >>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>>>>>>>
> >>>>>>>>>> Or you could use the tool lsof to find the open
> >>>>>>>>>> file descriptors.
> >>>>>>>>>>
> >>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>>>>>
> >>>>>>>>>> For both calls you should first change to the uid
> >>>>>>>>>> of the tomcat user or use sudo as in your
> >>>>>>>>>> example.
> >>>>>>>>>>
> >>>>>>>>>> Felix
> >>>>>>>>>>
> >>>>>>>>>>> Thanks and Regards Ayub
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher
> >>>>>>>>>>> Schultz < ch...@christopherschultz.net> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Ayub,
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>>>>>>>> Sorry we are using 8.0.32 version of
> >>>>>>>>>>>>>> tomcat.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> below is the configuration:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32
> >>>>>>>>>>>>>> (Ubuntu) Server built: Jan 24 2020
> >>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS
> >>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws
> >>>>>>>>>>>>>> Architecture: amd64 JVM Version:
> >>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor: Oracle
> >>>>>>>>>>>>>> Corporation
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I use the below command to check the
> >>>>>>>>>>>>>> file descriptors:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>>>>> So you know there is some kind of increase in
> >>>>>>>>>>> file-handle use, but you don't know what types
> >>>>>>>>>>> of file handles are increasing, right?
> >>>>>>>>>>>
> >>>>>>>>>>> Can you try to find out which kinds of file
> >>>>>>>>>>> handles are increasing?
> >>>>>>>>>>>
> >>>>>>>>>>> I have a sneaking suspicion that it's your
> >>>>>>>>>>> database connections and not actually files
> >>>>>>>>>>> open on the disk.
> >>>>>>>>>>>
> >>>>>>>>>>> Are you using a database connection pool? If
> >>>>>>>>>>> not, you should really use one and limit the
> >>>>>>>>>>> number of connections to something sane. If you
> >>>>>>>>>>> are using one, are you monitoring it to see how
> >>>>>>>>>>> many connections are actually being used? Are
> >>>>>>>>>>> you sure you are using proper resource
> >>>>>>>>>>> management[1]? Even a single code-path that
> >>>>>>>>>>> leaks connections can leak them quickly under
> >>>>>>>>>>> load.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> When there an issue related to broken
> >>>>>>>>>>>>>> files, this value keeps increasing, the
> >>>>>>>>>>>>>> only way to bring it down is to remove vm
> >>>>>>>>>>>>>> instance from AWS load balancer.> Which
> >>>>>>>>>>>>>> version of tomcat should I install ?
> >>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its
> >>>>>>>>>>> last release on 29 June 2018. That was 8.0.53.
> >>>>>>>>>>> Your release is from 8 February 2016 and is
> >>>>>>>>>>> dangerously out of date (unless you are using
> >>>>>>>>>>> the Ubuntu-packaged version, in which case I
> >>>>>>>>>>> hope they kept-up with security patches thee
> >>>>>>>>>>> past 4 years).
> >>>>>>>>>>>
> >>>>>>>>>>> -chris
> >>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
> >>>>>>>>>>>>>> Christopher Schultz <
> >>>>>>>>>>>>>> ch...@christopherschultz.net> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ayub,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases
> >>>>>>>>>>>>>>>>> open files on ubuntu AWS
> >>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If
> >>>>>>>>>>>>>> you aren't running the latest version
> >>>>>>>>>>>>>> (8.5.56), please upgrade and re-test.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If there is slow response from db I
> >>>>>>>>>>>>>>>>> see this stack trace and the open
> >>>>>>>>>>>>>>>>> files goes high and the only way to
> >>>>>>>>>>>>>>>>> open files go down is to remove the
> >>>>>>>>>>>>>>>>> instance from Amazon load
> >>>>>>>>>>>>>>>>> balancer.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is there a way to keep the open
> >>>>>>>>>>>>>>>>> files low even when Broken pipe
> >>>>>>>>>>>>>>>>> error is thrown ?
> >>>>>>>>>>>>>> What is your evidence that file handles
> >>>>>>>>>>>>>> are being left open?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Which file handles are being left open?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>> -------------------------------------------------------------
> - ---
> >
> >>>>>>>>>>
> - ---
> >>>>>
> >>>>>
> >>>>>>>>>>
> > ---------------------------------------------------------------------
> >>>>>
> >
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For
> >>>>> additional commands, e-mail: users-h...@tomcat.apache.org
> >>>>>
> >>>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8
> pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI
> rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD
> JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg
> 9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR
> nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p
> OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY
> Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz
> AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS
> tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G
> 7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa
> MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg=
> =q6xp
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
