I've got an amd64 web app server running 6.0/tomcat 7.0/jdk 1.8.
Occasionally I get a lot of FDs building up, looking like this in
fstat:
$ fstat | grep internet.stream.tcp | head -1500 |tail -5
_tomcat java 88950 1585* internet stream tcp 0x0 *:0
_tomcat java 88950 1586* internet stream tcp 0x0 *:0
_tomcat java 88950 1587* internet stream tcp 0x0 *:0
_tomcat java 88950 1588* internet stream tcp 0x0 *:0
_tomcat java 88950 1589* internet stream tcp 0x0 *:0
While I'm trying to figure out from the other side what's triggering
them, does anyone have an idea what these "*:0" entries are? They
don't show up in netstat.
If I capture earlier fstat output and match up the FD numbers they're
just outgoing connections from the web app (e.g. HTTP API fetches etc)
and at some point "something" happens and they change to the "*:0"
entries.
$ fstat | fgrep -c "internet stream tcp 0x0 *:0"
4678
$ fstat | grep -c internet.*tcp
5031
$ netstat -anfinet | grep -c tcp
570
$ netstat -anfinet | grep tcp | cut -c70- | sort | uniq -c
166 CLOSE_WAIT
126 ESTABLISHED
4 FIN_WAIT_2
23 LISTEN
251 TIME_WAIT
Any clues greatly appreciated!
OpenBSD 6.0 (GENERIC.MP) #2: Mon Oct 17 10:22:47 CEST 2016
r...@stable-60-amd64.mtier.org:/binpatchng/work-binpatch60-amd64/src/sys/arch/amd64/compile/GENERIC.MP
RTC BIOS diagnostic error 80<clock_battery>
real mem = 8562929664 (8166MB)
avail mem = 8298942464 (7914MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6720 (57 entries)
bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011
bios0: Dell Inc. PowerEdge R210 II
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR
HEST ERST BERT EINJ
acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5)
PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5)
PXSX(S4) RP06(S5) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.51 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.02 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 1, core 0, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.02 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.02 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 3 (P0P1)
acpiprt2 at acpi0: bus 2 (RP01)
acpiprt3 at acpi0: bus -1 (RP02)
acpiprt4 at acpi0: bus -1 (RP03)
acpiprt5 at acpi0: bus -1 (RP04)
acpiprt6 at acpi0: bus -1 (RP05)
acpiprt7 at acpi0: bus -1 (RP06)
acpiprt8 at acpi0: bus -1 (RP07)
acpiprt9 at acpi0: bus -1 (RP08)
acpiprt10 at acpi0: bus 1 (PEG0)
acpiprt11 at acpi0: bus -1 (PEG1)
acpiprt12 at acpi0: bus -1 (PEG2)
acpiprt13 at acpi0: bus -1 (PEG3)
acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1
mwait.1), PSS
acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1
mwait.1), PSS
acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1
mwait.1), PSS
acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1
mwait.1), PSS
acpipwrres0 at acpi0: FN00, resource for FAN0
acpipwrres1 at acpi0: FN01, resource for FAN1
acpipwrres2 at acpi0: FN02, resource for FAN2
acpipwrres3 at acpi0: FN03, resource for FAN3
acpipwrres4 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 100 degC
"IPI0001" at acpi0 not configured
"INT3F0D" at acpi0 not configured
"PNP0A05" at acpi0 not configured
"PNP0A05" at acpi0 not configured
"PNP0501" at acpi0 not configured
"PNP0501" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 3100 MHz: speeds: 3100, 2600, 2400, 2200, 2000, 1800,
1600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 2G Host" rev 0x09
ppb0 at pci0 dev 1 function 0 "Intel Core 2G PCIE" rev 0x09: msi
pci1 at ppb0 bus 1
mpii0 at pci1 dev 0 function 0 "Symbios Logic SAS2008" rev 0x03: msi
mpii0: PERC H200A, firmware 7.15.4.0 IR, MPI 2.0
scsibus1 at mpii0: 42 targets
sd0 at scsibus1 targ 1 lun 0: <Dell, Virtual Disk, 1028> SCSI4 0/direct fixed
naa.600508e000000000f1714e542fe1d004
sd0: 237824MB, 512 bytes/sector, 487063552 sectors
ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb1 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
pci2 at ppb1 bus 2
bnx0 at pci2 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
bnx1 at pci2 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
pci3 at ppb2 bus 3
3:3:0: mem address conflict 0xffff0000/0x10000
vga1 at pci3 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
ahci0: port 4: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 4 lun 0: <PLDS, DVD+-RW DS-8A5SH, XD51> ATAPI 5/cdrom
removable
ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19
iic0 at ichiic0
sdtemp0 at iic0 addr 0x19: stts2002
sdtemp1 at iic0 addr 0x1b: stts2002
spdmem0 at iic0 addr 0x51: 4GB DDR3 SDRAM ECC PC3-10600 with thermal sensor
spdmem1 at iic0 addr 0x53: 4GB DDR3 SDRAM ECC PC3-10600 with thermal sensor
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub2 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub3 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub4 at uhub3 port 5 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr
3
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
