-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Ayub,
On 6/24/20 11:05, Ayub Khan wrote:
> If some open file is owned by nginx why would it show up if I run
> the below command> sudo lsof -p $(cat /var/run/tomcat8.pid)
Because network connections have two ends.
- -chris
> On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> Ayub,
>
> On 6/23/20 19:17, Ayub Khan wrote:
>>>> Yes we have nginx as reverse proxy, below is the nginx
>>>> config. We notice this issue only when there is high number
>>>> of requests, during non peak hours we do not see this issue.>
>>>> location /myapp/myservice{ #local machine proxy_pass
>>>> http://localhost:8080; proxy_http_version 1.1;
>>>>
>>>> proxy_set_header Connection $connection_upgrade;
>>>> proxy_set_header Upgrade $http_upgrade;
>>>> proxy_set_header Host $host;
>>>> proxy_set_header X-Real-IP $remote_addr;
>>>> proxy_set_header X-Forwarded-For
>>>> $proxy_add_x_forwarded_for;
>>>>
>>>>
>>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>
> You might want to read about tuning nginx to drop connections after
> a certain period of time, number of requests, etc. Looks like
> either a bug in nginx or a misconfiguration which allows
> connections to stick-around like this. You may have to ask the
> nginx people. I have no experience with nginx myself, while others
> here may have some experience.
>
>>>> location / { # if using AWS Load balancer, this bit checks
>>>> for the presence of the https proto flag. if regular http is
>>>> found, then issue a redirect
> to hit
>>>> the https endpoint instead if ($http_x_forwarded_proto !=
>>>> 'https') { rewrite ^ https://$host$request_uri? permanent; }
>>>>
>>>> proxy_pass http://127.0.0.1:8080;
>>>> proxy_http_version 1.1;
>>>>
>>>> proxy_set_header Connection $connection_upgrade;
>>>> proxy_set_header Upgrade $http_upgrade;
>>>> proxy_set_header Host $host;
>>>> proxy_set_header X-Real-IP $remote_addr;
>>>> proxy_set_header X-Forwarded-For
>>>> $proxy_add_x_forwarded_for;
>>>>
>>>>
>>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>>>>
>>>> *below is the connector*
>>>>
>>>> <Connector port="8080"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>> connectionTimeout="2000" maxThreads="50000"
>>>> URIEncoding="UTF-8" redirectPort="8443" />
>
> 50k threads is a LOT of threads. Do you expect to handle 50k
> requests simultaneously?
>
>>>> these ports are random, I am not sure who owns the process.
>>>>
>>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port
>>>> 55866 is a random port.
> I'm sure you'll find that 55866 is owned by nginx. netstat will
> tell you .
>
> I think you need to look at your nginx configuration. It would also
> be a great time to upgrade to a supported version of Tomcat. I
> would recommend 8.5.56 or 9.0.36.
>
> -chris
>
>>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
>>>> ch...@christopherschultz.net> wrote:
>>>>
>>>> Ayub,
>>>>
>>>> On 6/23/20 16:23, Ayub Khan wrote:
>>>>>>> I executed *sudo lsof -p $(cat /var/run/tomcat8.pid)
>>>>>>> *and I saw the below output, some in CLOSE_WAIT and
>>>>>>> others in ESTABLISHED. If there are 200 open file
>>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count
>>>>>>> for CLOSE_WAIT increases I just have to restart
>>>>>>> tomcat.
>>>>>>>
>>>>>>> java 65189 tomcat8 715u IPv6
>>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760
>>>>>>> (CLOSE_WAIT) java 65189 tomcat8 716u IPv6
>>>>>>> 237848923 0t0 TCP
>>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>>>>
>>>> These are connections from some process into Tomcat listening
>>>> on port 8080 (that's what localhost:http-alt is). So what
>>>> process owns the outgoing connection on port 40568 on the
>>>> same host?
>>>>
>>>> Are you using a reverse proxy?
>>>>
>>>>>>> most of the open files are in CLOSE_WAIT state I do not
>>>>>>> see anything related to database ip.
>>>>
>>>> Agreed. It looks like you have a reverse proxy who is
>>>> losing-track of connections, or who is (re)opening
>>>> connections when it may be unnecessar y.
>>>>
>>>> Can you share your <Connector> configuration from
>>>> server.xml? Remember to remove any secrets.
>>>>
>>>> -chris
>>>>
>>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
>>>>>>> felix.schumac...@internetallee.de> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>>>>>>>>> Felix,
>>>>>>>>>
>>>>>>>>> I executed ls -l /proc/$(cat
>>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
>>>>>>>> output
>>>>>>>>> I see majority of them are related to sockets as
>>>>>>>>> shown below, some of
>>>>>>>> them
>>>>>>>>> point to the jar file of tomcat and others to the
>>>>>>>>> log file which is
>>>>>>>> created.
>>>>>>>>>
>>>>>>>>> socket:[2084570754] socket:[2084579487]
>>>>>>>>> socket:[2084578478] socket:[2084570167]
>>>>>>>>
>>>>>>>> Can you try the other command (lsof -p $(cat
>>>>>>>> ...tomcat.pid))? It should give a bit more details on
>>>>>>>> the used sockets that the proc directory.
>>>>>>>>
>>>>>>>> Felix
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>>>>>>>>> felix.schumac...@internetallee.de> wrote:
>>>>>>>>>
>>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>>>>>>>>>> Chris,
>>>>>>>>>>>
>>>>>>>>>>> I am using HikariCP for connection pooling. If
>>>>>>>>>>> the database is leaking connections then I
>>>>>>>>>>> should see connection not available exception.
>>>>>>>>>>>
>>>>>>>>>>> How do I find out which file descriptors are
>>>>>>>>>>> leaking ? these are not
>>>>>>>>>> files
>>>>>>>>>>> open on disk as there is no explicit disk file
>>>>>>>>>>> I/O in this application.
>>>>>>>>>>>
>>>>>>>>>>> I just use the below command to check for open
>>>>>>>>>>> file descriptors:
>>>>>>>>>>>
>>>>>>>>>>> watch "sudo ls /proc/`cat
>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>>>>>>>>>> You could have a look at the name of the files in
>>>>>>>>>> the pids proc
>>>>>>>> directory.
>>>>>>>>>>
>>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>>>>>>>
>>>>>>>>>> Or you could use the tool lsof to find the open
>>>>>>>>>> file descriptors.
>>>>>>>>>>
>>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>>>>>>>>>>
>>>>>>>>>> For both calls you should first change to the uid
>>>>>>>>>> of the tomcat user or use sudo as in your
>>>>>>>>>> example.
>>>>>>>>>>
>>>>>>>>>> Felix
>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards Ayub
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher
>>>>>>>>>>> Schultz < ch...@christopherschultz.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ayub,
>>>>>>>>>>>
>>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>>>>>>>>>>>>>> Sorry we are using 8.0.32 version of
>>>>>>>>>>>>>> tomcat.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> below is the configuration:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32
>>>>>>>>>>>>>> (Ubuntu) Server built: Jan 24 2020
>>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS
>>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws
>>>>>>>>>>>>>> Architecture: amd64 JVM Version:
>>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor: Oracle
>>>>>>>>>>>>>> Corporation
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I use the below command to check the
>>>>>>>>>>>>>> file descriptors:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> watch "sudo ls /proc/`cat
>>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>>>>>>>>>>> So you know there is some kind of increase in
>>>>>>>>>>> file-handle use, but you don't know what types
>>>>>>>>>>> of file handles are increasing, right?
>>>>>>>>>>>
>>>>>>>>>>> Can you try to find out which kinds of file
>>>>>>>>>>> handles are increasing?
>>>>>>>>>>>
>>>>>>>>>>> I have a sneaking suspicion that it's your
>>>>>>>>>>> database connections and not actually files
>>>>>>>>>>> open on the disk.
>>>>>>>>>>>
>>>>>>>>>>> Are you using a database connection pool? If
>>>>>>>>>>> not, you should really use one and limit the
>>>>>>>>>>> number of connections to something sane. If you
>>>>>>>>>>> are using one, are you monitoring it to see how
>>>>>>>>>>> many connections are actually being used? Are
>>>>>>>>>>> you sure you are using proper resource
>>>>>>>>>>> management[1]? Even a single code-path that
>>>>>>>>>>> leaks connections can leak them quickly under
>>>>>>>>>>> load.
>>>>>>>>>>>
>>>>>>>>>>>>>> When there an issue related to broken
>>>>>>>>>>>>>> files, this value keeps increasing, the
>>>>>>>>>>>>>> only way to bring it down is to remove vm
>>>>>>>>>>>>>> instance from AWS load balancer.> Which
>>>>>>>>>>>>>> version of tomcat should I install ?
>>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its
>>>>>>>>>>> last release on 29 June 2018. That was 8.0.53.
>>>>>>>>>>> Your release is from 8 February 2016 and is
>>>>>>>>>>> dangerously out of date (unless you are using
>>>>>>>>>>> the Ubuntu-packaged version, in which case I
>>>>>>>>>>> hope they kept-up with security patches thee
>>>>>>>>>>> past 4 years).
>>>>>>>>>>>
>>>>>>>>>>> -chris
>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
>>>>>>>>>>>>>> Christopher Schultz <
>>>>>>>>>>>>>> ch...@christopherschultz.net> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ayub,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases
>>>>>>>>>>>>>>>>> open files on ubuntu AWS
>>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If
>>>>>>>>>>>>>> you aren't running the latest version
>>>>>>>>>>>>>> (8.5.56), please upgrade and re-test.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If there is slow response from db I
>>>>>>>>>>>>>>>>> see this stack trace and the open
>>>>>>>>>>>>>>>>> files goes high and the only way to
>>>>>>>>>>>>>>>>> open files go down is to remove the
>>>>>>>>>>>>>>>>> instance from Amazon load
>>>>>>>>>>>>>>>>> balancer.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is there a way to keep the open
>>>>>>>>>>>>>>>>> files low even when Broken pipe
>>>>>>>>>>>>>>>>> error is thrown ?
>>>>>>>>>>>>>> What is your evidence that file handles
>>>>>>>>>>>>>> are being left open?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Which file handles are being left open?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -chris
>>>>>>>>>>>>>>>
>>>>>>>>>> -------------------------------------------------------------
- ---
>
>>>>>>>>>>
- ---
>>>>>
>>>>>
>>>>>>>>>>
> ---------------------------------------------------------------------
>>>>>
>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For
>>>>> additional commands, e-mail: users-h...@tomcat.apache.org
>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8
pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI
rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD
JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg
9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR
nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p
OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY
Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz
AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS
tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G
7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa
MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg=
=q6xp
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
