On 11/4/2012 12:38 μμ, Nikolaos Milas wrote:
> I continue to see this almost daily crash ("nfsen ... died for unknown
> reason)". Can anyone suggest a cause and a solution?
I've tried to find more details on the problem. So I tried to collect
some debug info (I'm not an expert) and in the log I see the following
about the first failed run (after about twelve hours):
-------------------------------------------------------------------------------------
Apr 20 21:30:29 monserver nfsen[6933]: connection on UNIX socket
Apr 20 21:30:29 monserver nfsen[6933]: comm server started: 18240
Apr 20 21:30:39 monserver nfsen[18240]: Failed: Timeout reading from
socket! at /data/nfsen/libexec/Nfcomm.pm line 1131.
Apr 20 21:30:39 monserver nfsen[6933]: comm child[18240] terminated with
no exit value
-------------------------------------------------------------------------------------
When running:
(It seems that the multiple nfsend-comm processes remain from previous runs of
nfsen; when nfsen dies "for unknown reason", I run nfsen start and nfsen starts
but an nfsend-comm from the previous run remains in the process list.)
# ps axjf
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
...
1 30577 30577 30577 ? -1 Ss 504 0:07
/data/nfsen/bin/nfsend-comm
1 21259 21259 21259 ? -1 Ss 504 0:01
/data/nfsen/bin/nfsend-comm
1 11816 11816 11816 ? -1 Ss 504 0:00
/data/nfsen/bin/nfsend-comm
1 19817 19817 19817 ? -1 Ss 504 0:01
/data/nfsen/bin/nfsend-comm
1 25822 25822 25822 ? -1 Ss 504 0:04
/data/nfsen/bin/nfsend-comm
1 2131 2131 2131 ? -1 Ss 504 0:03
/data/nfsen/bin/nfsend-comm
...
1 6932 6932 6932 ? -1 Rs 504 2:40 /usr/bin/perl -w
/data/nfsen/bin/nfsend
6932 6933 6933 6933 ? -1 Ss 504 0:00 \_
/data/nfsen/bin/nfsend-comm
# lsof -p 6932
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nfsend 6932 nfsen cwd DIR 252,0 4096 2 /
nfsend 6932 nfsen rtd DIR 252,0 4096 2 /
nfsend 6932 nfsen txt REG 252,0 19200 75700
/usr/bin/perl
nfsend 6932 nfsen mem REG 252,0 144776 1301763
/lib64/ld-2.5.so
nfsend 6932 nfsen mem REG 252,0 1722328 1301781
/lib64/libc-2.5.so
nfsend 6932 nfsen mem REG 252,0 23360 1301798
/lib64/libdl-2.5.so
nfsend 6932 nfsen mem REG 252,0 615136 1301995
/lib64/libm-2.5.so
nfsend 6932 nfsen mem REG 252,0 145872 1301799
/lib64/libpthread-2.5.so
nfsend 6932 nfsen mem REG 252,0 85544 1301997
/lib64/libz.so.1.2.3
nfsend 6932 nfsen mem REG 252,0 53448 1301800
/lib64/librt-2.5.so
nfsend 6932 nfsen mem REG 252,0 51280 73941
/usr/lib64/libdbi.so.0.0.5
nfsend 6932 nfsen mem REG 252,0 690576 78964
/usr/lib64/librrd.so.4.1.4
nfsend 6932 nfsen mem REG 252,0 1262480 228801
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
nfsend 6932 nfsen mem REG 252,0 12040 72516
/usr/lib64/libXau.so.6.0.0
nfsend 6932 nfsen mem REG 252,0 22032 72673
/usr/lib64/libXdmcp.so.6.0.0
nfsend 6932 nfsen mem REG 252,0 1099816 72985
/usr/lib64/libX11.so.6.2.0
nfsend 6932 nfsen mem REG 252,0 262904 1301946
/lib64/libgobject-2.0.so.0.1200.3
nfsend 6932 nfsen mem REG 252,0 114352 1302051
/lib64/libnsl-2.5.so
nfsend 6932 nfsen mem REG 252,0 13520 1301945
/lib64/libgmodule-2.0.so.0.1200.3
nfsend 6932 nfsen mem REG 252,0 92816 1302056
/lib64/libresolv-2.5.so
nfsend 6932 nfsen mem REG 252,0 143144 1302011
/lib64/libexpat.so.0.5.0
nfsend 6932 nfsen mem REG 252,0 543824 69252
/usr/lib64/libfreetype.so.6.3.10
nfsend 6932 nfsen mem REG 252,0 38528 73002
/usr/lib64/libXrender.so.1.3.0
nfsend 6932 nfsen mem REG 252,0 212896 69259
/usr/lib64/libfontconfig.so.1.1.0
nfsend 6932 nfsen mem REG 252,0 432992 73825
/usr/lib64/libcairo.so.2.9.2
nfsend 6932 nfsen mem REG 252,0 149344 70222
/usr/lib64/libpng12.so.0.10.0
nfsend 6932 nfsen mem REG 252,0 1297136 76276
/usr/lib64/libxml2.so.2.6.26
nfsend 6932 nfsen mem REG 252,0 39304 74098
/usr/lib64/libpangocairo-1.0.so.0.1400.9
nfsend 6932 nfsen mem REG 252,0 192152 73833
/usr/lib64/libpangoft2-1.0.so.0.1400.9
nfsend 6932 nfsen mem REG 252,0 265536 69216
/usr/lib64/libpango-1.0.so.0.1400.9
nfsend 6932 nfsen mem REG 252,0 18152 1302075
/lib64/libutil-2.5.so
nfsend 6932 nfsen mem REG 252,0 48600 1302080
/lib64/libcrypt-2.5.so
nfsend 6932 nfsen mem REG 252,0 25920 2701809
/var/cache/fontconfig/beeeeb3dfe132a8a0633a017c99ce0c0-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 56414128 69193
/usr/lib/locale/locale-archive
nfsend 6932 nfsen mem REG 252,0 119224 229644
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
nfsend 6932 nfsen mem REG 252,0 21424 229774
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
nfsend 6932 nfsen mem REG 252,0 14840 229057
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
nfsend 6932 nfsen mem REG 252,0 15880 229770
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so
nfsend 6932 nfsen mem REG 252,0 31656 229457
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
nfsend 6932 nfsen mem REG 252,0 10800 228792
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
nfsend 6932 nfsen mem REG 252,0 37400 229771
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
nfsend 6932 nfsen mem REG 252,0 97334 619478
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so
nfsend 6932 nfsen mem REG 252,0 44016 2701801
/var/cache/fontconfig/e19de935dec46bbf3ed114ee4965548a-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 10136 2701251
/var/cache/fontconfig/e3ead4b767b8819993a6fa3ae306afa9-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 647608 1301942
/lib64/libglib-2.0.so.0.1200.3
nfsend 6932 nfsen mem REG 252,0 13056 228836
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
nfsend 6932 nfsen mem REG 252,0 81800 229709
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
nfsend 6932 nfsen mem REG 252,0 18080 228851
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
nfsend 6932 nfsen mem REG 252,0 28544 2701803
/var/cache/fontconfig/7ddba6133ef499da58de5e8c586d3b75-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 53880 1301825
/lib64/libnss_files-2.5.so
nfsend 6932 nfsen mem REG 252,0 66360 2701404
/var/cache/fontconfig/73a61b34dd8ca4d8a159807604ab432f-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 32184 2701806
/var/cache/fontconfig/b79f3aaa7d385a141ab53ec885cc22a8-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 7184 2701804
/var/cache/fontconfig/87f5e051180a7a75f16eb6fe7dbd3749-x86-64.cache-2
nfsend 6932 nfsen mem REG 252,0 11704 358189
/usr/lib64/pango/1.5.0/modules/pango-basic-fc.so
nfsend 6932 nfsen mem REG 252,0 237540 618666
/usr/share/fonts/dejavu-lgc/DejaVuLGCSansMono.ttf
nfsend 6932 nfsen 0r CHR 1,3 0t0 1204 /dev/null
nfsend 6932 nfsen 1w CHR 1,3 0t0 1204 /dev/null
nfsend 6932 nfsen 2w CHR 1,3 0t0 1204 /dev/null
nfsend 6932 nfsen 3u unix 0xffff8100a7cadc40 0t0 10902452 socket
-------------------------------------------------------------------------------------
I also tried to run gdb at the same time:
# gdb /data/nfsen/bin/nfsend 6932
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
"/data/nfsen/bin/nfsend": not in executable format: File format not
recognized
Attaching to process 6932
Reading symbols from /usr/bin/perl...(no debugging symbols found)...done.
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libutil.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Sys/Syslog/Syslog.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IPC/SysV/SysV.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
Reading symbols from
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so...done.
Loaded symbols for
/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/RRDs/RRDs.so
Reading symbols from /usr/lib64/librrd.so.4...done.
Loaded symbols for /usr/lib64/librrd.so.4
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libdbi.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libdbi.so.0
Reading symbols from /usr/lib64/libpng12.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpng12.so.0
Reading symbols from /usr/lib64/libpangocairo-1.0.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libpangocairo-1.0.so.0
Reading symbols from /usr/lib64/libpango-1.0.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libpango-1.0.so.0
Reading symbols from /usr/lib64/libcairo.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcairo.so.2
Reading symbols from /lib64/libgobject-2.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgobject-2.0.so.0
Reading symbols from /lib64/libgmodule-2.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgmodule-2.0.so.0
Reading symbols from /lib64/libglib-2.0.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libglib-2.0.so.0
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libpangoft2-1.0.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libpangoft2-1.0.so.0
Reading symbols from /usr/lib64/libfontconfig.so.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libfontconfig.so.1
Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfreetype.so.6
Reading symbols from /usr/lib64/libXrender.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libXrender.so.1
Reading symbols from /usr/lib64/libX11.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libX11.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libexpat.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libexpat.so.0
Reading symbols from /usr/lib64/libXau.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libXau.so.6
Reading symbols from /usr/lib64/libXdmcp.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libXdmcp.so.6
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Storable/Storable.so
Reading symbols from
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from
/usr/lib64/pango/1.5.0/modules/pango-basic-fc.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/pango/1.5.0/modules/pango-basic-fc.so
warning: no loadable sections found in added symbol-file system-supplied
DSO at 0x7fff95ffd000
0x000000372a09a8d0 in __nanosleep_nocancel () from /lib64/libc.so.6
(gdb) continue
Continuing.
Detaching after fork from child process 7558.
Detaching after fork from child process 7559.
...
<Many identical messages: Detaching after fork from child process xxxxx.>
...
Detaching after fork from child process 18133.
Detaching after fork from child process 18135.
Program received signal SIGPIPE, Broken pipe.
0x000000372ac0df15 in send () from /lib64/libpthread.so.0
-------------------------------------------------------------------------------------
Does that help at all?
If you would like me to further debug, please provide details on what
should I do.
Please advise.
Thanks,
Nick
------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
