Re: what does "internet stream tcp 0x0 *:0" mean?
On Thu, May 04, 2017 at 03:03:21PM +0100, Stuart Henderson wrote: > On 2017/05/04 15:59, Alexander Bluhm wrote: > > On Thu, May 04, 2017 at 02:29:14PM +0100, Stuart Henderson wrote: > > > The information that netstat can show for these isn't particularly > > > useful, I think adding them to the netstat output would mainly just > > > increase noise. > > > > Yes. > > > > > They matter more when under FD pressure, people will already look > > > at fstat in that case (which can give them better information). > > > Do we want something like this in fstat(1)? > > > > Yes please. That will help people who wonder what all these 0x0 > > mean. And perhaps another sentence that answers you original > > question. > > > > "Sockets that have been disassociated from a protocol control block > > will always print 0x0." > > Thanks, I was trying to think of nice wording for this - I think > that's perfect. > > jmc, how does that look to you? > i'm fine with it. there is similar wording in fstat(1) regarding info available to the superuser, so you may want to tune it a little. the options -o and -s have: This information is only visible to the user or superuser. so, for example, "visible" might be better than "available", but up to you. jmc > Index: fstat.1 > === > RCS file: /cvs/src/usr.bin/fstat/fstat.1,v > retrieving revision 1.54 > diff -u -p -r1.54 fstat.1 > --- fstat.1 21 Jan 2017 12:21:56 - 1.54 > +++ fstat.1 4 May 2017 14:02:47 - > @@ -217,6 +217,9 @@ For example, the addresses mentioned abo > .Ic netstat -A > command would print for TCP, UDP, and > .Ux Ns -domain. > +These addresses are only available to the superuser, otherwise 0x0 is > printed. > +Sockets that have been disassociated from a protocol control block > +will always print 0x0. > A unidirectional > .Ux Ns -domain > socket indicates the direction of flow with >
Re: what does "internet stream tcp 0x0 *:0" mean?
On 2017/05/04 15:59, Alexander Bluhm wrote: > On Thu, May 04, 2017 at 02:29:14PM +0100, Stuart Henderson wrote: > > The information that netstat can show for these isn't particularly > > useful, I think adding them to the netstat output would mainly just > > increase noise. > > Yes. > > > They matter more when under FD pressure, people will already look > > at fstat in that case (which can give them better information). > > Do we want something like this in fstat(1)? > > Yes please. That will help people who wonder what all these 0x0 > mean. And perhaps another sentence that answers you original > question. > > "Sockets that have been disassociated from a protocol control block > will always print 0x0." Thanks, I was trying to think of nice wording for this - I think that's perfect. jmc, how does that look to you? Index: fstat.1 === RCS file: /cvs/src/usr.bin/fstat/fstat.1,v retrieving revision 1.54 diff -u -p -r1.54 fstat.1 --- fstat.1 21 Jan 2017 12:21:56 - 1.54 +++ fstat.1 4 May 2017 14:02:47 - @@ -217,6 +217,9 @@ For example, the addresses mentioned abo .Ic netstat -A command would print for TCP, UDP, and .Ux Ns -domain. +These addresses are only available to the superuser, otherwise 0x0 is printed. +Sockets that have been disassociated from a protocol control block +will always print 0x0. A unidirectional .Ux Ns -domain socket indicates the direction of flow with
Re: what does "internet stream tcp 0x0 *:0" mean?
On Thu, May 04, 2017 at 02:29:14PM +0100, Stuart Henderson wrote: > The information that netstat can show for these isn't particularly > useful, I think adding them to the netstat output would mainly just > increase noise. Yes. > They matter more when under FD pressure, people will already look > at fstat in that case (which can give them better information). > Do we want something like this in fstat(1)? Yes please. That will help people who wonder what all these 0x0 mean. And perhaps another sentence that answers you original question. "Sockets that have been disassociated from a protocol control block will always print 0x0." > > Index: fstat.1 > === > RCS file: /cvs/src/usr.bin/fstat/fstat.1,v > retrieving revision 1.54 > diff -u -p -r1.54 fstat.1 > --- fstat.1 21 Jan 2017 12:21:56 - 1.54 > +++ fstat.1 4 May 2017 13:28:49 - > @@ -217,6 +217,7 @@ For example, the addresses mentioned abo > .Ic netstat -A > command would print for TCP, UDP, and > .Ux Ns -domain. > +These addresses are only available to the superuser, otherwise 0x0 is > printed. > A unidirectional > .Ux Ns -domain > socket indicates the direction of flow with
Re: Fwd: what does "internet stream tcp 0x0 *:0" mean?
On 2017/05/04 14:45, Andrew Grillet wrote: > "The socket has no TCP PCB anymore. " > > This would be a better error message. It's more a statement of fact rather than an error, and the other information in this line is important for tracking it down. > (Although I don't know what a PCB is, > and when I Google, it comes out as PolyChlorinated Biphenyl or Printed > Circuit Board, > so perhaps a bit more explanation is needed). Protocol Control Block - not explicitly described in fstat(1) but it is in netstat(1), and those who can do something useful with the information will already know what it means.
Fwd: what does "internet stream tcp 0x0 *:0" mean?
"The socket has no TCP PCB anymore. "
This would be a better error message. (Although I don't know what a PCB is,
and when I Google, it comes out as PolyChlorinated Biphenyl or Printed
Circuit Board,
so perhaps a bit more explanation is needed).
-- Forwarded message --
From: Alexander Bluhm <alexander.bl...@gmx.net>
Date: 4 May 2017 at 09:45
Subject: Re: what does "internet stream tcp 0x0 *:0" mean?
To: tech@openbsd.org
On Wed, May 03, 2017 at 02:22:23PM +0100, Stuart Henderson wrote:
> On 2017/05/03 10:33, Stuart Henderson wrote:
> > $ fstat | grep internet.stream.tcp | head -1500 |tail -5
> > _tomcat java 88950 1585* internet stream tcp 0x0 *:0
The socket has no TCP PCB anymore. This can happen either after
a TCP reset packet or a shutdown and a TCP time wait timeout. The
attached Perl script triggers both cases.
I consider it as a bug that these connection don't appear in netstat.
bluhm
#!/usr/bin/perl
# create stream sockets without tcp pcb
use strict;
use warnings;
use IO::Socket::INET;
use Socket;
$> == 0
or die "Must run as root for netstat and fstat pointers";
my $ls = IO::Socket::INET->new(
Domain=> AF_INET,
LocalAddr => "127.0.0.1",
Proto => "tcp",
Listen=> 1,
) or die "Create listen socket failed: $!";
my $port = $ls->sockport();
# connect and reset
my $cs = IO::Socket::INET->new(
Domain=> AF_INET,
PeerAddr => "127.0.0.1",
PeerPort => $port,
Proto => "tcp",
) or die "Create connect socket failed: $!";
my $as = $ls->accept()
or die "Accept socket failed: $!";
$cs->setsockopt(SOL_SOCKET, SO_LINGER, pack('ii', 1, 0))
or die "Linger connect socket failed: $!";
$cs->close()
or die "Close connect socket failed: $!";
sleep 1;
print "\nReset:\n";
print map { " $_" } `netstat -Aan -f inet -p tcp`;
print map { " $_" } grep { /internet stream tcp/ } `fstat -p $$`;
# connect and shutdown
$cs = IO::Socket::INET->new(
Domain=> AF_INET,
PeerAddr => "127.0.0.1",
PeerPort => $port,
Proto => "tcp",
) or die "Create connect socket failed: $!";
$as = $ls->accept()
or die "Accept socket failed: $!";
$as->shutdown(SHUT_WR)
or die "Shutdown accept socket failed: $!";
$cs->close()
or die "Close connect socket failed: $!";
# TIME WAIT Timeout=2MSL + 1
sleep 61;
print "\nShutdown:\n";
print map { " $_" } `netstat -Aan -f inet -p tcp`;
print map { " $_" } grep { /internet stream tcp/ } `fstat -p $$`;
Re: what does "internet stream tcp 0x0 *:0" mean?
On 2017/05/04 10:50, Alexander Bluhm wrote: > On Thu, May 04, 2017 at 10:45:27AM +0200, Alexander Bluhm wrote: > > I consider it as a bug that these connection don't appear in netstat. > > Not so sure anymore whether this is a bug. The diff shows those > sockets also in netstat. > > 0x0 tcp 0 0 *.**.*CLOSED > root perl 712496* internet stream tcp 0x0 *:0 > > But there is nothing in network layer associated with them anymore. > Is netstat the right place for them to be displayed? The information that netstat can show for these isn't particularly useful, I think adding them to the netstat output would mainly just increase noise. They matter more when under FD pressure, people will already look at fstat in that case (which can give them better information). Do we want something like this in fstat(1)? Index: fstat.1 === RCS file: /cvs/src/usr.bin/fstat/fstat.1,v retrieving revision 1.54 diff -u -p -r1.54 fstat.1 --- fstat.1 21 Jan 2017 12:21:56 - 1.54 +++ fstat.1 4 May 2017 13:28:49 - @@ -217,6 +217,7 @@ For example, the addresses mentioned abo .Ic netstat -A command would print for TCP, UDP, and .Ux Ns -domain. +These addresses are only available to the superuser, otherwise 0x0 is printed. A unidirectional .Ux Ns -domain socket indicates the direction of flow with
Re: what does "internet stream tcp 0x0 *:0" mean?
On Thu, May 04, 2017 at 10:45:27AM +0200, Alexander Bluhm wrote:
> I consider it as a bug that these connection don't appear in netstat.
Not so sure anymore whether this is a bug. The diff shows those
sockets also in netstat.
0x0 tcp 0 0 *.**.*CLOSED
root perl 712496* internet stream tcp 0x0 *:0
But there is nothing in network layer associated with them anymore.
Is netstat the right place for them to be displayed?
bluhm
Index: usr.bin/netstat/inet.c
===
RCS file: /data/mirror/openbsd/cvs/src/usr.bin/netstat/inet.c,v
retrieving revision 1.156
diff -u -p -r1.156 inet.c
--- usr.bin/netstat/inet.c 7 Mar 2017 23:35:06 - 1.156
+++ usr.bin/netstat/inet.c 4 May 2017 08:35:49 -
@@ -192,11 +192,11 @@ protopr(kvm_t *kvmd, u_long pcbaddr, u_i
kf[i].unp_path[0] != '\0'))
if ((af == AF_LOCAL || af == AF_UNSPEC) && !proto)
unixdomainpr([i]);
- if (kf[i].so_family == AF_INET && kf[i].so_pcb != 0 &&
+ if (kf[i].so_family == AF_INET &&
kf[i].inp_rtableid == tableid)
if (af == AF_INET || af == AF_UNSPEC)
netdomainpr([i], proto);
- if (kf[i].so_family == AF_INET6 && kf[i].so_pcb != 0 &&
+ if (kf[i].so_family == AF_INET6 &&
kf[i].inp_rtableid == tableid)
if (af == AF_INET6 || af == AF_UNSPEC)
netdomainpr([i], proto);
Index: sys/kern/kern_sysctl.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/kern/kern_sysctl.c,v
retrieving revision 1.325
diff -u -p -r1.325 kern_sysctl.c
--- sys/kern/kern_sysctl.c 27 Apr 2017 21:39:27 - 1.325
+++ sys/kern/kern_sysctl.c 4 May 2017 08:37:14 -
@@ -1326,10 +1326,14 @@ sysctl_file(int *name, u_int namelen, ch
if (fp->f_count > 1 && /* 0, +1 for our FREF() */
(arg == 0 || fp->f_type == arg)) {
int af, skip = 0;
+ void *pcb;
if (arg == DTYPE_SOCKET && fp->f_type == arg) {
af = ((struct socket *)fp->f_data)->
so_proto->pr_domain->dom_family;
- if (af == AF_INET || af == AF_INET6)
+ pcb = ((struct socket *)fp->f_data)->
+ so_pcb;
+ if ((af == AF_INET || af == AF_INET6)
+ && pcb != NULL)
skip = 1;
}
if (!skip)
Re: what does "internet stream tcp 0x0 *:0" mean?
On Wed, May 03, 2017 at 02:22:23PM +0100, Stuart Henderson wrote:
> On 2017/05/03 10:33, Stuart Henderson wrote:
> > $ fstat | grep internet.stream.tcp | head -1500 |tail -5
> > _tomcat java 88950 1585* internet stream tcp 0x0 *:0
The socket has no TCP PCB anymore. This can happen either after
a TCP reset packet or a shutdown and a TCP time wait timeout. The
attached Perl script triggers both cases.
I consider it as a bug that these connection don't appear in netstat.
bluhm
#!/usr/bin/perl
# create stream sockets without tcp pcb
use strict;
use warnings;
use IO::Socket::INET;
use Socket;
$> == 0
or die "Must run as root for netstat and fstat pointers";
my $ls = IO::Socket::INET->new(
Domain=> AF_INET,
LocalAddr => "127.0.0.1",
Proto => "tcp",
Listen=> 1,
) or die "Create listen socket failed: $!";
my $port = $ls->sockport();
# connect and reset
my $cs = IO::Socket::INET->new(
Domain=> AF_INET,
PeerAddr => "127.0.0.1",
PeerPort => $port,
Proto => "tcp",
) or die "Create connect socket failed: $!";
my $as = $ls->accept()
or die "Accept socket failed: $!";
$cs->setsockopt(SOL_SOCKET, SO_LINGER, pack('ii', 1, 0))
or die "Linger connect socket failed: $!";
$cs->close()
or die "Close connect socket failed: $!";
sleep 1;
print "\nReset:\n";
print map { " $_" } `netstat -Aan -f inet -p tcp`;
print map { " $_" } grep { /internet stream tcp/ } `fstat -p $$`;
# connect and shutdown
$cs = IO::Socket::INET->new(
Domain=> AF_INET,
PeerAddr => "127.0.0.1",
PeerPort => $port,
Proto => "tcp",
) or die "Create connect socket failed: $!";
$as = $ls->accept()
or die "Accept socket failed: $!";
$as->shutdown(SHUT_WR)
or die "Shutdown accept socket failed: $!";
$cs->close()
or die "Close connect socket failed: $!";
# TIME WAIT Timeout=2MSL + 1
sleep 61;
print "\nShutdown:\n";
print map { " $_" } `netstat -Aan -f inet -p tcp`;
print map { " $_" } grep { /internet stream tcp/ } `fstat -p $$`;
what does "internet stream tcp 0x0 *:0" mean?
I've got an amd64 web app server running 6.0/tomcat 7.0/jdk 1.8. Occasionally I get a lot of FDs building up, looking like this in fstat: $ fstat | grep internet.stream.tcp | head -1500 |tail -5 _tomcat java 88950 1585* internet stream tcp 0x0 *:0 _tomcat java 88950 1586* internet stream tcp 0x0 *:0 _tomcat java 88950 1587* internet stream tcp 0x0 *:0 _tomcat java 88950 1588* internet stream tcp 0x0 *:0 _tomcat java 88950 1589* internet stream tcp 0x0 *:0 While I'm trying to figure out from the other side what's triggering them, does anyone have an idea what these "*:0" entries are? They don't show up in netstat. If I capture earlier fstat output and match up the FD numbers they're just outgoing connections from the web app (e.g. HTTP API fetches etc) and at some point "something" happens and they change to the "*:0" entries. $ fstat | fgrep -c "internet stream tcp 0x0 *:0" 4678 $ fstat | grep -c internet.*tcp 5031 $ netstat -anfinet | grep -c tcp 570 $ netstat -anfinet | grep tcp | cut -c70- | sort | uniq -c 166 CLOSE_WAIT 126 ESTABLISHED 4 FIN_WAIT_2 23 LISTEN 251 TIME_WAIT Any clues greatly appreciated! OpenBSD 6.0 (GENERIC.MP) #2: Mon Oct 17 10:22:47 CEST 2016 r...@stable-60-amd64.mtier.org:/binpatchng/work-binpatch60-amd64/src/sys/arch/amd64/compile/GENERIC.MP RTC BIOS diagnostic error 80 real mem = 8562929664 (8166MB) avail mem = 8298942464 (7914MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe6720 (57 entries) bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011 bios0: Dell Inc. PowerEdge R210 II acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR HEST ERST BERT EINJ acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.51 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 100MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.02 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 1, core 0, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.02 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, 3100.02 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,POPCNT,DEADLINE,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 1, package 0 ioapic0 at mainbus0: apid 0 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 addr 0xe000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 3 (P0P1) acpiprt2 at acpi0: bus 2 (RP01) acpiprt3 at acpi0: bus -1 (RP02) acpiprt4 at acpi0: bus -1 (RP03) acpiprt5 at acpi0: bus -1 (RP04) acpiprt6 at acpi0: bus -1 (RP05) acpiprt7 at acpi0: bus -1 (RP06) acpiprt8 at acpi0: bus -1 (RP07) acpiprt9 at acpi0: bus -1 (RP08) acpiprt10 at acpi0: bus 1 (PEG0) acpiprt11 at acpi0: bus -1 (PEG1) acpiprt12 at acpi0: bus -1 (PEG2) acpiprt13 at acpi0: bus -1 (PEG3) acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpipwrres0 at
