I wrote a script that checks the FD counts for every java pid on the server.
Just looking at these results, I don't think we're hitting an FD limit at the
moment, but I'll try it again tomorrow while the problem is presenting.
[root@app51b scripts]# for s in $(ps ax|grep java|awk '{print $1}');do
fd_counts $s;echo -e "----------\n";done
Process ID: 784860
Max open files: 16384
Used file descriptors: 631
Available file descriptors: 15753
----------
Process ID: 801288
Max open files: 16384
Used file descriptors: 70
Available file descriptors: 16314
----------
Process ID: 809263
Max open files: 16384
Used file descriptors: 78
Available file descriptors: 16306
----------
Process ID: 827276
Max open files: 16384
Used file descriptors: 549
Available file descriptors: 15835
----------
Process ID: 847407
Max open files: 16384
Used file descriptors: 94
Available file descriptors: 16290
----------
Process ID: 862359
Max open files: 16384
Used file descriptors: 605
Available file descriptors: 15779
----------
Process ID: 877635
Max open files: 16384
Used file descriptors: 82
Available file descriptors: 16302
----------
Process ID: 887797
Max open files: 16384
Used file descriptors: 102
Available file descriptors: 16282
----------
Process ID: 898698
Max open files: 16384
Used file descriptors: 54
Available file descriptors: 16330
----------
Process ID: 903756
Max open files: 16384
Used file descriptors: 665
Available file descriptors: 15719
----------
Process ID: 918138
Max open files: 16384
Used file descriptors: 457
Available file descriptors: 15927
----------
Process ID: 929377
Max open files: 16384
Used file descriptors: 99
Available file descriptors: 16285
----------
Process ID: 940403
Max open files: 16384
Used file descriptors: 614
Available file descriptors: 15770
----------
Error: Process with PID 946105 does not exist.
----------
Process ID: 955351
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 963602
Max open files: 16384
Used file descriptors: 82
Available file descriptors: 16302
----------
Process ID: 971272
Max open files: 16384
Used file descriptors: 95
Available file descriptors: 16289
----------
Process ID: 987470
Max open files: 16384
Used file descriptors: 73
Available file descriptors: 16311
----------
Process ID: 1002039
Max open files: 16384
Used file descriptors: 399
Available file descriptors: 15985
----------
Process ID: 1012180
Max open files: 16384
Used file descriptors: 117
Available file descriptors: 16267
----------
Process ID: 1021450
Max open files: 16384
Used file descriptors: 532
Available file descriptors: 15852
----------
Process ID: 1038105
Max open files: 16384
Used file descriptors: 61
Available file descriptors: 16323
----------
Process ID: 1047269
Max open files: 16384
Used file descriptors: 735
Available file descriptors: 15649
----------
Process ID: 1069422
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 1080743
Max open files: 16384
Used file descriptors: 49
Available file descriptors: 16335
----------
Process ID: 1087538
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 1091023
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 1098912
Max open files: 16384
Used file descriptors: 388
Available file descriptors: 15996
----------
Process ID: 1106941
Max open files: 16384
Used file descriptors: 606
Available file descriptors: 15778
----------
Process ID: 1124436
Max open files: 16384
Used file descriptors: 70
Available file descriptors: 16314
----------
Process ID: 1133799
Max open files: 16384
Used file descriptors: 73
Available file descriptors: 16311
----------
Process ID: 1141993
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 1154539
Max open files: 16384
Used file descriptors: 488
Available file descriptors: 15896
----------
Process ID: 1167799
Max open files: 16384
Used file descriptors: 224
Available file descriptors: 16160
----------
Process ID: 1175722
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 1185333
Max open files: 16384
Used file descriptors: 421
Available file descriptors: 15963
----------
Process ID: 1199328
Max open files: 16384
Used file descriptors: 108
Available file descriptors: 16276
----------
Process ID: 1214320
Max open files: 16384
Used file descriptors: 468
Available file descriptors: 15916
----------
Process ID: 1223773
Max open files: 16384
Used file descriptors: 113
Available file descriptors: 16271
----------
Process ID: 1233052
Max open files: 16384
Used file descriptors: 56
Available file descriptors: 16328
----------
Process ID: 1239823
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 1252744
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 1264677
Max open files: 16384
Used file descriptors: 494
Available file descriptors: 15890
----------
Process ID: 1274481
Max open files: 16384
Used file descriptors: 59
Available file descriptors: 16325
----------
Process ID: 1280662
Max open files: 16384
Used file descriptors: 61
Available file descriptors: 16323
----------
Process ID: 1286971
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 1303458
Max open files: 16384
Used file descriptors: 99
Available file descriptors: 16285
----------
Process ID: 1320163
Max open files: 16384
Used file descriptors: 82
Available file descriptors: 16302
----------
Process ID: 1332334
Max open files: 16384
Used file descriptors: 631
Available file descriptors: 15753
----------
Process ID: 1352329
Max open files: 16384
Used file descriptors: 112
Available file descriptors: 16272
----------
Process ID: 1369075
Max open files: 16384
Used file descriptors: 90
Available file descriptors: 16294
----------
Process ID: 1379011
Max open files: 16384
Used file descriptors: 140
Available file descriptors: 16244
----------
Process ID: 1394238
Max open files: 16384
Used file descriptors: 640
Available file descriptors: 15744
----------
Process ID: 1410817
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 1421164
Max open files: 16384
Used file descriptors: 93
Available file descriptors: 16291
----------
Process ID: 1436160
Max open files: 16384
Used file descriptors: 61
Available file descriptors: 16323
----------
Process ID: 1444706
Max open files: 16384
Used file descriptors: 593
Available file descriptors: 15791
----------
Process ID: 1463116
Max open files: 16384
Used file descriptors: 61
Available file descriptors: 16323
----------
Process ID: 1469155
Max open files: 16384
Used file descriptors: 111
Available file descriptors: 16273
----------
Process ID: 1479311
Max open files: 16384
Used file descriptors: 91
Available file descriptors: 16293
----------
Process ID: 1489252
Max open files: 16384
Used file descriptors: 66
Available file descriptors: 16318
----------
Process ID: 1497856
Max open files: 16384
Used file descriptors: 612
Available file descriptors: 15772
----------
Process ID: 1512364
Max open files: 16384
Used file descriptors: 119
Available file descriptors: 16265
----------
Process ID: 1523654
Max open files: 16384
Used file descriptors: 65
Available file descriptors: 16319
----------
Process ID: 1530872
Max open files: 16384
Used file descriptors: 385
Available file descriptors: 15999
----------
Process ID: 1539667
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 1548878
Max open files: 16384
Used file descriptors: 560
Available file descriptors: 15824
----------
Process ID: 1563260
Max open files: 16384
Used file descriptors: 50
Available file descriptors: 16334
----------
Process ID: 1568596
Max open files: 16384
Used file descriptors: 611
Available file descriptors: 15773
----------
Process ID: 1584177
Max open files: 16384
Used file descriptors: 386
Available file descriptors: 15998
----------
Process ID: 1593491
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 1605573
Max open files: 16384
Used file descriptors: 56
Available file descriptors: 16328
----------
Process ID: 1612474
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 1619567
Max open files: 16384
Used file descriptors: 567
Available file descriptors: 15817
----------
Process ID: 1638065
Max open files: 16384
Used file descriptors: 97
Available file descriptors: 16287
----------
Process ID: 1652157
Max open files: 16384
Used file descriptors: 70
Available file descriptors: 16314
----------
Process ID: 1666413
Max open files: 16384
Used file descriptors: 91
Available file descriptors: 16293
----------
Process ID: 1676748
Max open files: 16384
Used file descriptors: 59
Available file descriptors: 16325
----------
Process ID: 1683743
Max open files: 16384
Used file descriptors: 510
Available file descriptors: 15874
----------
Process ID: 1696232
Max open files: 16384
Used file descriptors: 564
Available file descriptors: 15820
----------
Process ID: 1718318
Max open files: 16384
Used file descriptors: 73
Available file descriptors: 16311
----------
Process ID: 1728316
Max open files: 16384
Used file descriptors: 408
Available file descriptors: 15976
----------
Process ID: 1736762
Max open files: 16384
Used file descriptors: 86
Available file descriptors: 16298
----------
Process ID: 1751411
Max open files: 16384
Used file descriptors: 96
Available file descriptors: 16288
----------
Process ID: 1758978
Max open files: 16384
Used file descriptors: 68
Available file descriptors: 16316
----------
Process ID: 1768254
Max open files: 16384
Used file descriptors: 95
Available file descriptors: 16289
----------
Process ID: 1783325
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 1795868
Max open files: 16384
Used file descriptors: 98
Available file descriptors: 16286
----------
Process ID: 1806547
Max open files: 16384
Used file descriptors: 79
Available file descriptors: 16305
----------
Process ID: 1818034
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 1830142
Max open files: 16384
Used file descriptors: 105
Available file descriptors: 16279
----------
Process ID: 1847550
Max open files: 16384
Used file descriptors: 80
Available file descriptors: 16304
----------
Process ID: 1858130
Max open files: 16384
Used file descriptors: 80
Available file descriptors: 16304
----------
Process ID: 1864744
Max open files: 16384
Used file descriptors: 78
Available file descriptors: 16306
----------
Process ID: 1871244
Max open files: 16384
Used file descriptors: 99
Available file descriptors: 16285
----------
Process ID: 1886362
Max open files: 16384
Used file descriptors: 86
Available file descriptors: 16298
----------
Process ID: 1897304
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 1903278
Max open files: 16384
Used file descriptors: 69
Available file descriptors: 16315
----------
Process ID: 1908274
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 1914139
Max open files: 16384
Used file descriptors: 96
Available file descriptors: 16288
----------
Process ID: 1924627
Max open files: 16384
Used file descriptors: 564
Available file descriptors: 15820
----------
Process ID: 1939261
Max open files: 16384
Used file descriptors: 114
Available file descriptors: 16270
----------
Process ID: 1957931
Max open files: 16384
Used file descriptors: 81
Available file descriptors: 16303
----------
Process ID: 1969576
Max open files: 16384
Used file descriptors: 393
Available file descriptors: 15991
----------
Process ID: 2000386
Max open files: 16384
Used file descriptors: 140
Available file descriptors: 16244
----------
Process ID: 2011495
Max open files: 16384
Used file descriptors: 108
Available file descriptors: 16276
----------
Process ID: 2035151
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 2041685
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 2048517
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 2054726
Max open files: 16384
Used file descriptors: 99
Available file descriptors: 16285
----------
Process ID: 2070775
Max open files: 16384
Used file descriptors: 77
Available file descriptors: 16307
----------
Process ID: 2078423
Max open files: 16384
Used file descriptors: 335
Available file descriptors: 16049
----------
Process ID: 2087105
Max open files: 16384
Used file descriptors: 70
Available file descriptors: 16314
----------
Process ID: 2093706
Max open files: 16384
Used file descriptors: 103
Available file descriptors: 16281
----------
Process ID: 2107447
Max open files: 16384
Used file descriptors: 488
Available file descriptors: 15896
----------
Process ID: 2118262
Max open files: 16384
Used file descriptors: 249
Available file descriptors: 16135
----------
Process ID: 2130982
Max open files: 16384
Used file descriptors: 86
Available file descriptors: 16298
----------
Process ID: 2141323
Max open files: 16384
Used file descriptors: 97
Available file descriptors: 16287
----------
Process ID: 2155514
Max open files: 16384
Used file descriptors: 77
Available file descriptors: 16307
----------
Process ID: 2170621
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 2182750
Max open files: 16384
Used file descriptors: 391
Available file descriptors: 15993
----------
Process ID: 2190310
Max open files: 16384
Used file descriptors: 106
Available file descriptors: 16278
----------
Process ID: 2201009
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 2209596
Max open files: 16384
Used file descriptors: 93
Available file descriptors: 16291
----------
Process ID: 2221845
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 2229102
Max open files: 16384
Used file descriptors: 79
Available file descriptors: 16305
----------
Process ID: 2243939
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2254308
Max open files: 16384
Used file descriptors: 86
Available file descriptors: 16298
----------
Process ID: 2264927
Max open files: 16384
Used file descriptors: 107
Available file descriptors: 16277
----------
Process ID: 2276583
Max open files: 16384
Used file descriptors: 77
Available file descriptors: 16307
----------
Process ID: 2292264
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2309238
Max open files: 16384
Used file descriptors: 104
Available file descriptors: 16280
----------
Process ID: 2321124
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2329568
Max open files: 16384
Used file descriptors: 89
Available file descriptors: 16295
----------
Process ID: 2341296
Max open files: 16384
Used file descriptors: 109
Available file descriptors: 16275
----------
Process ID: 2358431
Max open files: 16384
Used file descriptors: 494
Available file descriptors: 15890
----------
Process ID: 2368000
Max open files: 16384
Used file descriptors: 538
Available file descriptors: 15846
----------
Process ID: 2381261
Max open files: 16384
Used file descriptors: 79
Available file descriptors: 16305
----------
Process ID: 2400199
Max open files: 16384
Used file descriptors: 550
Available file descriptors: 15834
----------
Process ID: 2415660
Max open files: 16384
Used file descriptors: 104
Available file descriptors: 16280
----------
Process ID: 2433230
Max open files: 16384
Used file descriptors: 104
Available file descriptors: 16280
----------
Process ID: 2443947
Max open files: 16384
Used file descriptors: 77
Available file descriptors: 16307
----------
Process ID: 2454585
Max open files: 16384
Used file descriptors: 73
Available file descriptors: 16311
----------
Process ID: 2465052
Max open files: 16384
Used file descriptors: 581
Available file descriptors: 15803
----------
Process ID: 2479584
Max open files: 16384
Used file descriptors: 589
Available file descriptors: 15795
----------
Process ID: 2495809
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2507139
Max open files: 16384
Used file descriptors: 57
Available file descriptors: 16327
----------
Process ID: 2515217
Max open files: 16384
Used file descriptors: 90
Available file descriptors: 16294
----------
Process ID: 2528948
Max open files: 16384
Used file descriptors: 557
Available file descriptors: 15827
----------
Process ID: 2549749
Max open files: 16384
Used file descriptors: 80
Available file descriptors: 16304
----------
Process ID: 2564417
Max open files: 16384
Used file descriptors: 89
Available file descriptors: 16295
----------
Process ID: 2578973
Max open files: 16384
Used file descriptors: 594
Available file descriptors: 15790
----------
Process ID: 2595126
Max open files: 16384
Used file descriptors: 92
Available file descriptors: 16292
----------
Process ID: 2605048
Max open files: 16384
Used file descriptors: 609
Available file descriptors: 15775
----------
Process ID: 2620903
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2630370
Max open files: 16384
Used file descriptors: 464
Available file descriptors: 15920
----------
Process ID: 2641272
Max open files: 16384
Used file descriptors: 498
Available file descriptors: 15886
----------
Process ID: 2653304
Max open files: 16384
Used file descriptors: 81
Available file descriptors: 16303
----------
Process ID: 2675569
Max open files: 16384
Used file descriptors: 70
Available file descriptors: 16314
----------
Process ID: 2690820
Max open files: 16384
Used file descriptors: 632
Available file descriptors: 15752
----------
Process ID: 2710254
Max open files: 16384
Used file descriptors: 79
Available file descriptors: 16305
----------
Process ID: 2727041
Max open files: 16384
Used file descriptors: 60
Available file descriptors: 16324
----------
Process ID: 2735476
Max open files: 16384
Used file descriptors: 57
Available file descriptors: 16327
----------
Process ID: 2743989
Max open files: 16384
Used file descriptors: 91
Available file descriptors: 16293
----------
Process ID: 2764829
Max open files: 16384
Used file descriptors: 91
Available file descriptors: 16293
----------
Process ID: 2776309
Max open files: 16384
Used file descriptors: 83
Available file descriptors: 16301
----------
Process ID: 2798074
Max open files: 16384
Used file descriptors: 123
Available file descriptors: 16261
----------
Process ID: 2809094
Max open files: 16384
Used file descriptors: 71
Available file descriptors: 16313
----------
Process ID: 2829152
Max open files: 16384
Used file descriptors: 63
Available file descriptors: 16321
----------
Process ID: 2837183
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2850792
Max open files: 16384
Used file descriptors: 411
Available file descriptors: 15973
----------
Process ID: 2860142
Max open files: 16384
Used file descriptors: 85
Available file descriptors: 16299
----------
Process ID: 2869356
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 2878071
Max open files: 16384
Used file descriptors: 550
Available file descriptors: 15834
----------
Process ID: 2892918
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 2904643
Max open files: 16384
Used file descriptors: 594
Available file descriptors: 15790
----------
Process ID: 2922073
Max open files: 16384
Used file descriptors: 611
Available file descriptors: 15773
----------
Process ID: 2936770
Max open files: 16384
Used file descriptors: 103
Available file descriptors: 16281
----------
Process ID: 2948656
Max open files: 16384
Used file descriptors: 573
Available file descriptors: 15811
----------
Process ID: 2965304
Max open files: 16384
Used file descriptors: 121
Available file descriptors: 16263
----------
Process ID: 2980532
Max open files: 16384
Used file descriptors: 602
Available file descriptors: 15782
----------
Process ID: 2995116
Max open files: 16384
Used file descriptors: 93
Available file descriptors: 16291
----------
Process ID: 3004215
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 3015134
Max open files: 16384
Used file descriptors: 81
Available file descriptors: 16303
----------
Process ID: 3030897
Max open files: 16384
Used file descriptors: 71
Available file descriptors: 16313
----------
Process ID: 3057260
Max open files: 16384
Used file descriptors: 489
Available file descriptors: 15895
----------
Process ID: 3069905
Max open files: 16384
Used file descriptors: 79
Available file descriptors: 16305
----------
Process ID: 3087427
Max open files: 16384
Used file descriptors: 417
Available file descriptors: 15967
----------
Process ID: 3096167
Max open files: 16384
Used file descriptors: 549
Available file descriptors: 15835
----------
Process ID: 3113547
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 3124478
Max open files: 16384
Used file descriptors: 548
Available file descriptors: 15836
----------
Process ID: 3141399
Max open files: 16384
Used file descriptors: 269
Available file descriptors: 16115
----------
Process ID: 3162039
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 3183259
Max open files: 16384
Used file descriptors: 74
Available file descriptors: 16310
----------
Process ID: 3192511
Max open files: 16384
Used file descriptors: 555
Available file descriptors: 15829
----------
Process ID: 3211282
Max open files: 16384
Used file descriptors: 118
Available file descriptors: 16266
----------
Process ID: 3222776
Max open files: 16384
Used file descriptors: 84
Available file descriptors: 16300
----------
Process ID: 3234000
Max open files: 16384
Used file descriptors: 452
Available file descriptors: 15932
----------
Process ID: 3247214
Max open files: 16384
Used file descriptors: 78
Available file descriptors: 16306
----------
Process ID: 3257892
Max open files: 16384
Used file descriptors: 68
Available file descriptors: 16316
----------
Process ID: 3272553
Max open files: 16384
Used file descriptors: 57
Available file descriptors: 16327
----------
Process ID: 3281216
Max open files: 16384
Used file descriptors: 577
Available file descriptors: 15807
----------
Process ID: 3299720
Max open files: 16384
Used file descriptors: 533
Available file descriptors: 15851
----------
Process ID: 3313519
Max open files: 16384
Used file descriptors: 84
Available file descriptors: 16300
----------
Process ID: 3325713
Max open files: 16384
Used file descriptors: 79
Available file descriptors: 16305
----------
Process ID: 3343415
Max open files: 16384
Used file descriptors: 869
Available file descriptors: 15515
----------
Process ID: 3361366
Max open files: 16384
Used file descriptors: 68
Available file descriptors: 16316
----------
Process ID: 3377779
Max open files: 16384
Used file descriptors: 68
Available file descriptors: 16316
----------
Process ID: 3393952
Max open files: 16384
Used file descriptors: 67
Available file descriptors: 16317
----------
Process ID: 3406562
Max open files: 16384
Used file descriptors: 600
Available file descriptors: 15784
----------
Process ID: 3424586
Max open files: 16384
Used file descriptors: 591
Available file descriptors: 15793
----------
Process ID: 3439580
Max open files: 16384
Used file descriptors: 592
Available file descriptors: 15792
----------
Process ID: 3456032
Max open files: 16384
Used file descriptors: 106
Available file descriptors: 16278
----------
Process ID: 3468450
Max open files: 16384
Used file descriptors: 581
Available file descriptors: 15803
----------
Process ID: 3481288
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 3491853
Max open files: 16384
Used file descriptors: 97
Available file descriptors: 16287
----------
Process ID: 3506851
Max open files: 16384
Used file descriptors: 76
Available file descriptors: 16308
----------
Process ID: 3614221
Max open files: 16384
Used file descriptors: 75
Available file descriptors: 16309
----------
Process ID: 3920027
Max open files: 16384
Used file descriptors: 2047
Available file descriptors: 14337
----------
> -----Original Message-----
> From: Chuck Caldarale <n82...@gmail.com>
> Sent: Monday, June 24, 2024 2:17 PM
> To: Tomcat Users List <users@tomcat.apache.org>
> Subject: Re: Isolating the Root Cause of "Connection Refused"
>
>
> > On Jun 24, 2024, at 15:47, Eric Robinson <eric.robin...@psmnv.com> wrote:
> >
> >> -----Original Message-----
> >> From: Chuck Caldarale <n82...@gmail.com>
> >> Sent: Monday, June 24, 2024 1:40 PM
> >> To: Tomcat Users List <users@tomcat.apache.org>
> >> Subject: Re: Isolating the Root Cause of "Connection Refused"
> >>
> >>
> >>> On Jun 24, 2024, at 15:36, Eric Robinson <eric.robin...@psmnv.com>
> wrote:
> >>>
> >>>> -----Original Message-----
> >>>> From: Chuck Caldarale <n82...@gmail.com>
> >>>> Sent: Monday, June 24, 2024 1:29 PM
> >>>> To: Tomcat Users List <users@tomcat.apache.org>
> >>>> Subject: Re: Isolating the Root Cause of "Connection Refused"
> >>>>
> >>>>
> >>>>> On Jun 24, 2024, at 15:19, Eric Robinson <eric.robin...@psmnv.com>
> >> wrote:
> >>>>>
> >>>>> We have a tomcat server that is not that busy. It has 100 tomcat
> >>>>> instances
> >>>> running, but it handles a few hundred connections per second total,
> >>>> across all of them. It intermittently rejects connection attempts
> >>>> to listening tomcats. The server is running Rocky 8, has 48 cores
> >>>> (about 15-40% utilized), 1T RAM (400G free), with NVME storage.
> >>>> 'sar' shows
> >> almost 0% iowait.
> >>>>>
> >>>>> During production:
> >>>>>
> >>>>> * /proc/sys/net/netfilter/nf_conntrack_count shows anywhere from
> 100K
> >> to
> >>>> 250K connections
> >>>>> * /proc/sys/net/netfilter/nf_conntrack_max is set to 2M.
> >>>>> * netstat -an|wc -l usually shows 90-150K connections
> >>>>>
> >>>>> Obviously, the TCP stack must be running into some resource
> >>>>> limitation, or
> >>>> some kind of race condition. I've been working the issue for hours
> >>>> and days, without success. How can I determine exactly why the
> >>>> tomcats intermittently reject connections?
> >>>>
> >>>>
> >>>> Perhaps some of the Tomcat processes are occasionally running out
> >>>> of file descriptors?
> >>>>
> >>>
> >>> Great thought. Wouldn't tomcat log a message somewhere if that were
> >>> the
> >> case?
> >>
> >>
> >> No - Tomcat would never see the request and would have no knowledge
> >> that the OS blocked the connection attempt.
> >>
> >
> > But the OS should log something, I assume? I don't see anything in dmesg or
> messages.
>
>
> As Thomas noted, the OS won’t log this, since it’s considered to be an
> application error.
>
> You can look at /proc/<pid>/limits to see what any process of interest is
> limited
> to. Counting open files for a process is a bit trickier, but something like:
>
> ls -1 /proc/<pid>/fd | wc -l
>
> will do it. I don’t know of any commonly available tool to watch for open
> files
> getting close to the limit.
>
> You can experiment with larger values of acceptCount on your <Connector>
> elements to see if that might allow for better handling of bursts of
> connection
> requests. Also, it used to be that socket FDs were not released until garbage
> collection ran, but I’m not sure if that’s still the case in current versions
> of
> Tomcat and JVM.
>
> - Chuck
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
