The problem occurred again starting last night. I have another clue, but I still don't know how it is occurring or how to fix it.
It looks like all the TCP ports are in "bound" state, but not being released. How can I isolate the cause of this? # netstat -an UDP: IPv4 Local Address Remote Address State -------------------- -------------------- ---------- *.* Unbound *.57921 Idle *.42045 Idle *.* Unbound *.* Unbound *.* Unbound *.* Unbound *.33757 Idle *.37883 Idle *.40744 Idle *.4045 Idle *.111 Idle *.4045 Idle *.* Unbound *.60955 Idle *.42908 Idle *.56487 Idle *.111 Idle *.* Unbound *.60994 Idle *.* Unbound *.2049 Idle *.520 Idle *.2049 Idle *.46876 Idle *.50309 Idle *.123 Idle *.123 Idle 127.0.0.1.123 Idle 10.28.125.29.123 Idle *.57929 Idle *.64351 Idle *.63145 Idle *.39674 Idle *.65280 Idle *.52013 Idle *.47989 Idle *.* Unbound UDP: IPv6 Local Address Remote Address State If --------------------------------- --------------------------------- ---------- ----- *.57921 Idle *.* Unbound *.* Unbound *.40744 Idle *.111 Idle *.4045 Idle *.* Unbound *.60955 Idle *.42908 Idle *.* Unbound *.2049 Idle *.46876 Idle *.123 Idle ::1.123 Idle TCP: IPv4 Local Address Remote Address Swind Send-Q Rwind Recv-Q State -------------------- -------------------- ----- ------ ----- ------ ----------- *.60571 *.* 0 0 1057280 0 BOUND *.46344 *.* 0 0 1057280 0 BOUND *.39729 *.* 0 0 1057280 0 BOUND *.43531 *.* 0 0 1057280 0 BOUND *.49051 *.* 0 0 1057280 0 BOUND *.44876 *.* 0 0 1057280 0 BOUND *.65416 *.* 0 0 1057280 0 BOUND *.47714 *.* 0 0 1057280 0 BOUND *.59055 *.* 0 0 1057280 0 BOUND *.45033 *.* 0 0 1057280 0 BOUND *.63321 *.* 0 0 1057280 0 BOUND *.43896 *.* 0 0 1057280 0 BOUND *.46627 *.* 0 0 1057280 0 BOUND *.35555 *.* 0 0 1057280 0 BOUND *.36115 *.* 0 0 1057280 0 BOUND *.51969 *.* 0 0 1057280 0 BOUND *.63741 *.* 0 0 1057280 0 BOUND *.45747 *.* 0 0 1057280 0 BOUND *.33245 *.* 0 0 1057280 0 BOUND *.49925 *.* 0 0 1057280 0 BOUND *.63503 *.* 0 0 1057280 0 BOUND *.45319 *.* 0 0 1057280 0 BOUND *.39977 *.* 0 0 1057280 0 BOUND ....lots of lines deleted... *.37396 *.* 0 0 1057280 0 BOUND *.33735 *.* 0 0 1057280 0 BOUND *.35695 *.* 0 0 1057280 0 BOUND *.36589 *.* 0 0 1057280 0 BOUND *.41484 *.* 0 0 1057280 0 BOUND *.63428 *.* 0 0 1057280 0 BOUND *.54891 *.* 0 0 1057280 0 BOUND *.60222 *.* 0 0 1057280 0 BOUND *.40494 *.* 0 0 1057280 0 BOUND TCP: IPv6 Local Address Remote Address Swind Send-Q Rwind Recv-Q State If --------------------------------- --------------------------------- ----- ------ ----- ------ ----------- ----- *.54749 *.* 0 0 128000 0 LISTEN ::1.5999 *.* 0 0 128000 0 LISTEN *.4045 *.* 0 0 1049200 0 LISTEN *.111 *.* 0 0 128000 0 LISTEN *.* *.* 0 0 128000 0 IDLE *.45543 *.* 0 0 128000 0 LISTEN *.2049 *.* 0 0 1049200 0 LISTEN *.53926 *.* 0 0 128000 0 LISTEN *.50379 *.* 0 0 128000 0 LISTEN Active UNIX domain sockets Address Type Vnode Conn Local Addr Remote Addr ffffd063569210e8 stream-ord 0000000 0000000 ffffd06356921498 stream-ord 0000000 0000000 ffffd06356921bf8 stream-ord 0000000 0000000 ffffd063569260e0 stream-ord 0000000 0000000 ffffd06356926490 stream-ord ffffd0635646d300 0000000 private/scache ffffd06356926840 stream-ord 0000000 0000000 ffffd06356926bf0 stream-ord 0000000 0000000 ffffd063569290d8 stream-ord ffffd0635646d200 0000000 private/anvil ffffd06356929488 stream-ord 0000000 0000000 ffffd06356929838 stream-ord 0000000 0000000 ffffd06356929be8 stream-ord ffffd06356810480 0000000 private/lmtp ffffd0635692d0d0 stream-ord 0000000 0000000 ffffd0635692d480 stream-ord 0000000 0000000 ffffd0635692d830 stream-ord ffffd06356810280 0000000 private/virtual ffffd0635692dbe0 stream-ord 0000000 0000000 ffffd063569320c8 stream-ord 0000000 0000000 ffffd06356932478 stream-ord ffffd0635685aa00 0000000 private/local ffffd06356932828 stream-ord 0000000 0000000 ffffd06356932bd8 stream-ord 0000000 0000000 ffffd063569360c0 stream-ord ffffd0635685ad00 0000000 private/discard ffffd06356936470 stream-ord 0000000 0000000 ffffd06356936820 stream-ord 0000000 0000000 ffffd06356936bd0 stream-ord ffffd0635685ab00 0000000 private/retry ffffd0635693b0b8 stream-ord 0000000 0000000 ffffd0635693b468 stream-ord 0000000 0000000 ffffd0635693b818 stream-ord ffffd0635685ae00 0000000 private/error ffffd0635693bbc8 stream-ord 0000000 0000000 ffffd063568e10b0 stream-ord 0000000 0000000 ffffd063568e1460 stream-ord ffffd0635685a400 0000000 public/showq ffffd063568e1810 stream-ord 0000000 0000000 ffffd063568e1bc0 stream-ord 0000000 0000000 ffffd063568e60a8 stream-ord ffffd0635685a600 0000000 private/relay ffffd063568e6458 stream-ord 0000000 0000000 ffffd063568e6808 stream-ord 0000000 0000000 ffffd063568e6bb8 stream-ord ffffd0635685a900 0000000 private/smtp ffffd063568ea0a0 stream-ord 0000000 0000000 ffffd063568ea450 stream-ord 0000000 0000000 ffffd063568ea800 stream-ord ffffd0635646d100 0000000 private/proxywrite ffffd063568eabb0 stream-ord 0000000 0000000 ffffd063568f0098 stream-ord 0000000 0000000 ffffd063568f0448 stream-ord ffffd0635685ac00 0000000 private/proxymap ffffd063568f07f8 stream-ord 0000000 0000000 ffffd063568f0ba8 stream-ord 0000000 0000000 ffffd063568f2090 stream-ord ffffd0635685a200 0000000 public/flush ffffd063568f2440 stream-ord 0000000 0000000 ffffd063568f27f0 stream-ord 0000000 0000000 ffffd063568f2ba0 stream-ord ffffd0635685a500 0000000 private/verify ffffd063568f9088 stream-ord 0000000 0000000 ffffd063568f9438 stream-ord 0000000 0000000 ffffd063568f97e8 stream-ord ffffd06356810080 0000000 private/trace ffffd063568f9b98 stream-ord 0000000 0000000 ffffd063568fd080 stream-ord 0000000 0000000 ffffd063568fd430 stream-ord ffffd06356810180 0000000 private/defer ffffd063568fd7e0 stream-ord 0000000 0000000 ffffd063568fdb90 stream-ord 0000000 0000000 ffffd06356840078 stream-ord ffffd0635685a700 0000000 private/bounce ffffd06356840428 stream-ord 0000000 0000000 ffffd063568407d8 stream-ord 0000000 0000000 ffffd06356840b88 stream-ord ffffd0635685a800 0000000 private/rewrite ffffd06356843070 stream-ord ffffd06356810380 0000000 private/tlsmgr ffffd06356843420 stream-ord 0000000 0000000 ffffd063568437d0 stream-ord 0000000 0000000 ffffd06356849068 stream-ord 0000000 0000000 ffffd06356849418 stream-ord 0000000 0000000 ffffd063568497c8 stream-ord ffffd0635685a000 0000000 public/qmgr ffffd06356849b78 stream-ord ffffd0635685a100 0000000 public/cleanup ffffd0635684d060 stream-ord 0000000 0000000 ffffd0635684d410 stream-ord 0000000 0000000 ffffd0635684db70 stream-ord 0000000 0000000 ffffd06355646058 stream-ord 0000000 0000000 ffffd06355646b68 stream-ord ffffd0635685a300 0000000 public/pickup ffffd063551bf3f8 stream-ord ffffd063193fe900 0000000 /var/run/.inetd.uds ffffd063550e7b50 dgram ffffd063550eb380 0000000 /var/run/in.rdisc_mib ffffd06355031798 dgram ffffd063536c8800 0000000 /var/run/in.ndpd_mib ffffd06355031b48 stream-ord ffffd063536c8c00 0000000 /var/run/in.ndpd_ipadm ffffd0635265a028 stream-ord 0000000 ffffd0634e4acd00 /var/run/dbus/system_bus_socket ffffd0635265a788 stream-ord 0000000 ffffd063500ffc80 /var/run/hald/dbus-y1Me9kLIpf ffffd0635265ab38 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf ffffd06351d553d0 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf ffffd06351d55780 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf ffffd06351d55b30 stream-ord 0000000 ffffd063500ffc80 /var/run/hald/dbus-y1Me9kLIpf ffffd06351996018 stream-ord 0000000 ffffd063500ffc80 /var/run/hald/dbus-y1Me9kLIpf ffffd063519963c8 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf ffffd06351996778 stream-ord 0000000 ffffd063500ffc80 /var/run/hald/dbus-y1Me9kLIpf ffffd063500fe010 stream-ord 0000000 0000000 /var/run/hald/dbus-5Qrha0Wmu3 ffffd063500fe3c0 stream-ord 0000000 ffffd063500ffa80 /var/run/hald/dbus-5Qrha0Wmu3 ffffd063500fe770 stream-ord ffffd063500ffa80 0000000 /var/run/hald/dbus-5Qrha0Wmu3 ffffd063500feb20 stream-ord ffffd063500ffc80 0000000 /var/run/hald/dbus-y1Me9kLIpf ffffd0634e4ad008 stream-ord 0000000 0000000 ffffd0634e4ad3b8 stream-ord 0000000 0000000 ffffd0634e4ad768 stream-ord 0000000 0000000 /var/run/dbus/system_bus_socket ffffd0634e4adb18 stream-ord ffffd0634e4acd00 0000000 /var/run/dbus/system_bus_socket A sorted output shows nearly all 64K ports in bound state. On Tue, Jan 2, 2018 at 8:40 AM, Schweiss, Chip <c...@innovates.com> wrote: > About once every week or two I'm having NFS connections start to collapse > to one of my servers. Clients will lose thier connections of the the > course of several hours. The logs fill with these messages: > > Dec 25 16:21:14 mir-zfs03 rpcbind: [ID 452059 daemon.error] do_accept : > t_bind failed : Couldn't allocate address > Dec 25 16:21:14 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295 > daemon.error] t_bind(file descriptor 188/transport tcp) TLI error 5 > Dec 25 16:21:31 mir-zfs03 last message repeated 85 times > Dec 25 16:21:31 mir-zfs03 rpcbind: [ID 452059 daemon.error] do_accept : > t_bind failed : Couldn't allocate address > Dec 25 16:21:32 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295 > daemon.error] t_bind(file descriptor 188/transport tcp) TLI error 5 > Dec 25 16:21:34 mir-zfs03 last message repeated 19 times > Dec 25 16:21:37 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295 > daemon.error] t_bind(file descriptor 200/transport tcp) TLI error 5 > Dec 25 16:22:17 mir-zfs03 last message repeated 116 times > Dec 25 16:22:21 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295 > daemon.error] t_bind(file descriptor 206/transport tcp) TLI error 5 > Dec 25 16:23:04 mir-zfs03 last message repeated 81 times > > This is a fully updated OmniOS CE r151022. > > I've tried restarting NFS services, but the only thing that has been > successful in restoring services has been rebooting. > > I'm not finding anything useful via Google except the source code that > spits out this message. HP-UX appears to have had the same issue that > they patched years ago. I'm guessing shared NFS/RPC code. > > Any clue as to the cause of this and how to fix? > > -Chip > > >
_______________________________________________ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss