Ok so from that I can see that you're including:
include ("/usr/local/etc/conf.d/*.conf")
include('/etc/ganglia/conf.d/*.pyconf')Could you recheck what conf files you have in /usr/local/etc/conf.d/ ? Next thing - why are you building those packages without setting any proper (FHS like) directories (http://pl.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)? I'm almost sure that there is some configuration issue there On Sun, Feb 09, 2014 at 06:17:04PM -0500, Jeff Layton wrote: > Sure thing - I appreciate the help. > > Build options: > ./configure --with-gmetad > > gmond.conf: http://pastebin.com/ExiMgqv0 > > strace output: > I ran the strace using the following command: > > strace -s 1024 -ff -o strace.log gmond -d 5 -c /etc/ganglia/gmond.conf > > The output of the thread that has the segfault in it was uploaded > to pastebin: http://pastebin.com/xScMVU6P > > I had to erase the top 200 lines of the strace (too big and I'm not > a pro user - yet :) ). > > But... just to be sure, I'm attaching the compressed tarball. Apologies > to all but I just wanted to be sure. > > Once again - thanks a million! > > Jeff > > > > >Could you post here your build options (that ones you entered while > >./configure) and also could you paste gmond.conf into pastebin? > > > >Also plz strace one more time, but now with strace -s 1024 -e trace=file > >and paste the output to pastebin > > > > > >On Sun, Feb 09, 2014 at 04:55:14PM -0500, Jeff Layton wrote: > >>I hope this isn't too much output (I've heard about pastebin.com > >>but never really used it). > >> > >> > >>[root@home4 ganglia-3.6.0]# ldd /usr/local/sbin/gmond > >> linux-vdso.so.1 => (0x00007fff667f6000) > >> libapr-1.so.0 => /usr/lib64/libapr-1.so.0 (0x00007f6a24049000) > >> libresolv.so.2 => /lib64/libresolv.so.2 (0x000000337dc00000) > >> libganglia-3.6.0.so.0 => > >>/usr/local/lib64/libganglia-3.6.0.so.0 (0x00007f6a23e0d000) > >> libdl.so.2 => /lib64/libdl.so.2 (0x000000337c400000) > >> libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003390c00000) > >> libz.so.1 => /lib64/libz.so.1 (0x000000337d000000) > >> libpcre.so.0 => /lib64/libpcre.so.0 (0x0000003f73600000) > >> libexpat.so.1 => /lib64/libexpat.so.1 (0x000000337f800000) > >> libconfuse.so.0 => /usr/lib64/libconfuse.so.0 (0x00007f6a23bff000) > >> libpthread.so.0 => /lib64/libpthread.so.0 (0x000000337c800000) > >> libc.so.6 => /lib64/libc.so.6 (0x000000337c000000) > >> libuuid.so.1 => /lib64/libuuid.so.1 (0x0000003383800000) > >> libcrypt.so.1 => /lib64/libcrypt.so.1 (0x000000338c600000) > >> /lib64/ld-linux-x86-64.so.2 (0x000000337bc00000) > >> libfreebl3.so => /usr/lib64/libfreebl3.so (0x000000338ca00000) > >> > >> > >>Below is the tree output: > >> > >> > >>[root@home4 ganglia-3.6.0]# tree /etc/ganglia > >>/etc/ganglia > >>??? conf.d > >>? ??? procstat.pyconf > >>??? gmetad.conf > >>??? gmond.conf > >> > >>1 directory, 3 files > >> > >> > >>I looked at the strace file for process 3537 and I did see two places > >>where gmond does an access() on the python_modules directory. > >>Does gmond automatically look for the python modes so I don't need > >>to put them the modules section of gmond.conf? > >> > >>Thanks a million! > >> > >>Jeff > >> > >> > >>>Oh I didn't think about going that lowlevel :) Could you run ldd on > >>>gmond also? > >>> > >>>Could you also run 'tree' command on /etc/ganglia ? It's interesting > >>>that you have two times msg: "loaded module: python_module" while > >>>starting gmond. Rechecking this with strace log shows that it looks like > >>>double loading of those modules? http://pastebin.com/BjdCGgbj > >>> > >>> > >>>On Sun, Feb 09, 2014 at 03:28:10PM -0500, Jeff Layton wrote: > >>>>On 02/09/2014 02:48 PM, Jeff Layton wrote: > >>>>>On 02/09/2014 02:28 PM, Maciej Lasyk wrote: > >>>>> > >>>>>>You could also try to catch on which particular check this segfault > >>>>>>happens..? > >>>>>Not sure how to check this. When I run gmond interactively, it > >>>>>segfaults just after it says, > >>>>> > >>>>>[root@home4 yum.repos.d]# /usr/local/sbin/gmond -d 5 -c > >>>>>/etc/ganglia/gmond.conf > >>>>>loaded module: core_metrics > >>>>>loaded module: python_module > >>>>>loaded module: cpu_module > >>>>>loaded module: disk_module > >>>>>loaded module: load_module > >>>>>loaded module: mem_module > >>>>>loaded module: net_module > >>>>>loaded module: proc_module > >>>>>loaded module: sys_module > >>>>>loaded module: python_module > >>>>>Segmentation fault (core dumped) > >>>>> > >>>>> > >>>>>I'm not sure where to begin checking. I'm a very old-fashioned > >>>>>debugger - I tend to use a great deal of print statements to > >>>>>track down where things are happening. I can start doing this > >>>>>in gmond. > >>>>I tried putting fprintf's all over the gmond.c (yep - I'm that > >>>>poor of a debugger). I'm not sure but if looks like it segfaults > >>>>in the function setup_metric_callbacks on the statement, > >>>> > >>>> if (modp->init && modp->init(global_context)) { > >>>> > >>>>or on the function, > >>>> > >>>> apr_pool_cleanup_register(global_context, modp, > >>>> modular_metric_cleanup, > >>>> apr_pool_cleanup_null); > >>>> > >>>>I'm not too sure. > >>>> > >>>>I apologize if I'm wasting your time with my poor debugging > >>>>skills. > >>>> > >>>>Thanks! > >>>> > >>>>Jeff > >>>> > >>>> > >>>> > >>> > >>>------------------------------------------------------------------------------ > >>>Managing the Performance of Cloud-Based Applications > >>>Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > >>>Read the Whitepaper. > >>>http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk > >>> > >>> > >>>_______________________________________________ > >>>Ganglia-developers mailing list > >>>Ganglia-developers@lists.sourceforge.net > >>>https://lists.sourceforge.net/lists/listinfo/ganglia-developers > > > > > >------------------------------------------------------------------------------ > >Managing the Performance of Cloud-Based Applications > >Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > >Read the Whitepaper. > >http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk > > > > > >_______________________________________________ > >Ganglia-developers mailing list > >Ganglia-developers@lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/ganglia-developers > -- -- pozdrawiam, Maciej Lasyk GPG key ID: FFA8AEEC GPG info: http://maciek.lasyk.info/gpg.txt
pgpCYil1am6SO.pgp
Description: PGP signature
------------------------------------------------------------------------------ Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
