> On Jun 24, 2024, at 15:47, Eric Robinson <eric.robin...@psmnv.com> wrote:
>
>> -----Original Message-----
>> From: Chuck Caldarale <n82...@gmail.com>
>> Sent: Monday, June 24, 2024 1:40 PM
>> To: Tomcat Users List <users@tomcat.apache.org>
>> Subject: Re: Isolating the Root Cause of "Connection Refused"
>>
>>
>>> On Jun 24, 2024, at 15:36, Eric Robinson <eric.robin...@psmnv.com> wrote:
>>>
>>>> -----Original Message-----
>>>> From: Chuck Caldarale <n82...@gmail.com>
>>>> Sent: Monday, June 24, 2024 1:29 PM
>>>> To: Tomcat Users List <users@tomcat.apache.org>
>>>> Subject: Re: Isolating the Root Cause of "Connection Refused"
>>>>
>>>>
>>>>> On Jun 24, 2024, at 15:19, Eric Robinson <eric.robin...@psmnv.com>
>> wrote:
>>>>>
>>>>> We have a tomcat server that is not that busy. It has 100 tomcat
>>>>> instances
>>>> running, but it handles a few hundred connections per second total,
>>>> across all of them. It intermittently rejects connection attempts to
>>>> listening tomcats. The server is running Rocky 8, has 48 cores (about
>>>> 15-40% utilized), 1T RAM (400G free), with NVME storage. 'sar' shows
>> almost 0% iowait.
>>>>>
>>>>> During production:
>>>>>
>>>>> * /proc/sys/net/netfilter/nf_conntrack_count shows anywhere from 100K
>> to
>>>> 250K connections
>>>>> * /proc/sys/net/netfilter/nf_conntrack_max is set to 2M.
>>>>> * netstat -an|wc -l usually shows 90-150K connections
>>>>>
>>>>> Obviously, the TCP stack must be running into some resource
>>>>> limitation, or
>>>> some kind of race condition. I've been working the issue for hours
>>>> and days, without success. How can I determine exactly why the
>>>> tomcats intermittently reject connections?
>>>>
>>>>
>>>> Perhaps some of the Tomcat processes are occasionally running out of
>>>> file descriptors?
>>>>
>>>
>>> Great thought. Wouldn't tomcat log a message somewhere if that were the
>> case?
>>
>>
>> No - Tomcat would never see the request and would have no knowledge that the
>> OS blocked the connection attempt.
>>
>
> But the OS should log something, I assume? I don't see anything in dmesg or
> messages.
As Thomas noted, the OS won’t log this, since it’s considered to be an
application error.
You can look at /proc/<pid>/limits to see what any process of interest is
limited to. Counting open files for a process is a bit trickier, but something
like:
ls -1 /proc/<pid>/fd | wc -l
will do it. I don’t know of any commonly available tool to watch for open files
getting close to the limit.
You can experiment with larger values of acceptCount on your <Connector>
elements to see if that might allow for better handling of bursts of connection
requests. Also, it used to be that socket FDs were not released until garbage
collection ran, but I’m not sure if that’s still the case in current versions
of Tomcat and JVM.
- Chuck
