I'm trying to setup the tcp listener for rrdcached but getting this error
"absolute path names not allowed when talking to a remote daemon". Should I
be setting rrd_rootdir to something else besides defaults?
-gary
On Fri, Sep 12, 2014 at 3:12 PM, Devon H. O'Dell <devon.od...@gmail.com>
wrote:
> There's a lot less locking when rrdcached runs on tcp for various
> reasons. I switched to spinlocks because in the general case there
> aren't many metrics that people are interested in summarizing, which
> is why the CPU goes so high for you. I think that using
> (?:un)?summarized_metrics will significantly reduce your CPU usage due
> to reduction in time spent iterating hashmaps versus time spent
> writing to them.
>
> I intended to change the way this was done more extensively, but
> haven't yet had time.
>
> Dropbox (or similar) or web link would be fine for the core for me.
> You can also email it to me privately, but others on the list might
> also be interested if it's not sensitive.
>
> --dho
>
> 2014-09-12 14:41 GMT-07:00 Gary Barrueto <g...@justin.tv>:
> > Yes, I can provide the core. How should I share it? I'll see about
> setting
> > the (un)summarized_metrics options. Is there a good way to see which are
> > used or not?
> >
> > As for rrdcached this is how we have it running:
> >
> > /usr/bin/rrdcached -l unix:/var/run/rrdcached/sock -j
> > /var/lib/rrdcached/journal/ -F -P PENDING,FLUSH,STATS,HELP -b
> > /var/lib/ganglia/rrds -B -p /var/run/rrdcached/pid -g -z 150 -w 1800 -f
> 3600
> > -t 12
> >
> > export RRDCACHED_ADDRESS=unix:/var/run/rrdcached/sock ; /usr/sbin/gmetad
> -d1
> >
> > The server that gmetad/rrdcached/the 48 gmond processes (one for each
> data
> > source) runs on has dual L5640 processors. Will switch to rrdcached on
> tcp
> > also to see if that helps.
> >
> > -gary
> >
> > On Thu, Sep 11, 2014 at 6:52 PM, Vladimir Vuksan <vli...@veus.hr> wrote:
> >>
> >> If you can provide the core file of the segfault that would be helpful.
> >>
> >> Problem may be with metric summarization where it's locking. One thing
> to
> >> do is provide a list of metrics to summarize e.g. I got these options
> turned
> >> on in gmetad.conf
> >>
> >> summarized_metrics bytes_in bytes_out pkts_in pkts_out cpu_aidle
> cpu_idle
> >> cpu_intr cpu_nice cpu_num cpu_sintr cpu_speed cpu_steal cpu_system
> cpu_user
> >> cpu_wio
> >> summarized_metrics mem_buffers mem_cached mem_dirty mem_free mem_mapped
> >> mem_shared mem_total load_one
> >>
> >> as well as metrics that we are never gonna summarize e.g.
> >>
> >> unsummarized_metrics varnish rx tx ethtool mysql bird diskstat ipmi
> tcpext
> >> tcp et vm ipmi disk icmp procstat icmpmsg ap apssl ip management1
> >>
> >> Also can you tell me how your rrdcached is configured ? We have switched
> >> to TCP e.g. in gmetad.conf we have
> >>
> >> rrdcached_address 127.0.0.1:9998
> >>
> >> and rrdcached is started via
> >>
> >> /usr/bin/rrdcached -t 60 -w 360 -z 360 -F -s ganglia -m 664 -l
> >> 127.0.0.1:9998 -s ganglia -m 777 -P FLUSH,STATS,HELP -l
> >> unix:/tmp/rrdcached.limited.sock -b /var/lib/ganglia/rrds -B -p
> >> /var/lib/ganglia/rrdcached.pid -p /var/lib/ganglia/rrdcached.pid
> >>
> >> Let me know if this helps.
> >>
> >> Vladimir
> >> P.S. Our system does over 700k metrics with those settings
> >>
> >>
> >> On 09/10/2014 08:52 PM, Gary Barrueto wrote:
> >>
> >> We're seeing gmetad (version 3.7.0) threads segfaulting or gmetad and
> all
> >> its threads spinning 100% cpu. The system is running ubuntu
> 12.04/precise.
> >> Our ganglia has 48 data source, 1300 hosts and has around 220k metrics.
> We
> >> are running this with rrdcached. Would like to fix it from segfault and
> spin
> >> on cpu. I'm including a backtrace of when gmetad and threads were
> spinning
> >> on cpu.
> >>
> >> Is there anything I can provide to help figure this out?
> >>
> >> gmetad.conf looks like this:
> >>
> >> RRAs "RRA:AVERAGE:0.5:1:5856" "RRA:AVERAGE:0.5:24:1680"
> >> "RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:274"
> >> "RRA:AVERAGE:0.5:5760:3740"
> >> server_threads 10
> >> case_sensitive_hostnames 1
> >>
> >> and includes 48 data_source tags and each has 2 data sources.
> >>
> >> ganglia was ./configured using:
> >>
> >> ./configure --host=x86_64-linux-gnu --build=x86_64-linux-gnu
> --prefix=/usr
> >> --mandir=${prefix}/share/man --libdir=${prefix}/lib
> >> --sysconfdir=/etc/ganglia --infodir=${prefix}/share/info --enable-shared
> >> --with-gmetad --with-memcached --enable-debug
> >>
> >> Backtrace:
> >> GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
> >> Copyright (C) 2012 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later
> >> <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "x86_64-linux-gnu".
> >> For bug reporting instructions, please see:
> >> <http://bugs.launchpad.net/gdb-linaro/>...
> >> Reading symbols from /usr/sbin/gmetad...done.
> >> [New LWP 26916]
> >> [New LWP 26917]
> >> [New LWP 26918]
> >> [New LWP 26919]
> >> [New LWP 26920]
> >> [New LWP 26921]
> >> [New LWP 26922]
> >> [New LWP 26923]
> >> [New LWP 26924]
> >> [New LWP 26925]
> >> [New LWP 26926]
> >> [New LWP 26927]
> >> [New LWP 26928]
> >> [New LWP 26929]
> >> [New LWP 26930]
> >> [New LWP 26931]
> >> [New LWP 26932]
> >> [New LWP 26933]
> >> [New LWP 26934]
> >> [New LWP 26935]
> >> [New LWP 26936]
> >> [New LWP 26937]
> >> [New LWP 26938]
> >> [New LWP 26939]
> >> [New LWP 26940]
> >> [New LWP 26941]
> >> [New LWP 26942]
> >> [New LWP 26943]
> >> [New LWP 26944]
> >> [New LWP 26946]
> >> [New LWP 26947]
> >> [New LWP 26948]
> >> [New LWP 26949]
> >> [New LWP 26950]
> >> [New LWP 26951]
> >> [New LWP 26952]
> >> [New LWP 26953]
> >> [New LWP 26954]
> >> [New LWP 26955]
> >> [New LWP 26956]
> >> [New LWP 26957]
> >> [New LWP 26958]
> >> [New LWP 26959]
> >> [New LWP 26960]
> >> [New LWP 26961]
> >> [New LWP 26962]
> >> [New LWP 26963]
> >> [New LWP 26964]
> >> [New LWP 26965]
> >> [New LWP 26966]
> >> [New LWP 26967]
> >> [New LWP 26968]
> >> [New LWP 26969]
> >> [New LWP 26970]
> >> [New LWP 26974]
> >> [New LWP 26975]
> >> [New LWP 26976]
> >> [New LWP 26977]
> >> [New LWP 26978]
> >> [New LWP 26979]
> >> [New LWP 26980]
> >> [New LWP 26981]
> >> [New LWP 26982]
> >> [New LWP 26983]
> >> [New LWP 26908]
> >> [Thread debugging using libthread_db enabled]
> >> Using host libthread_db library
> "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >> Core was generated by `gmetad'.
> >> #0 __lll_lock_wait () at
> >> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> >> 132 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such
> >> file or directory.
> >> (gdb) thread apply all bt
> >>
> >> Thread 65 (LWP 26908):
> >> #0 0x00007f316bed2436 in ck_pr_stall () at
> >> /usr/include/gcc/x86_64/ck_pr.h:70
> >> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:163
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x4057d1
> >> <do_root_summary>, arg=0x0) at hash.c:417
> >> #3 0x00000000004064f5 in main (argc=2, argv=0x7fff547ded18) at
> >> gmetad.c:586
> >>
> >> Thread 64 (LWP 26983):
> >> #0 0x00007f316bed2533 in ck_pr_fas_uint (target=0x7f30b402c230, v=1) at
> >> /usr/include/gcc/x86_64/ck_pr.h:115
> >> #1 0x00007f316bed4376 in ck_rwlock_write_lock (rw=0x7f30b402c230) at
> >> /usr/include/ck_rwlock.h:115
> >> #2 0x00007f316bed528f in hash_delete (key=0x7f30b48792e0,
> >> hash=0x7f30b40f4240) at hash.c:335
> >> #3 0x00007f316bed4a75 in hash_destroy (hash=0x7f30b40f4240) at
> hash.c:127
> >> #4 0x0000000000424d47 in cleanup_source (key=0x7f30b49dd210,
> >> val=0x7f30b49dd140, arg=0x7f30c77fdca0) at cleanup.c:170
> >> #5 0x00007f316bed54c0 in hash_walkfrom (hash=0x1e23420, from=0,
> >> func=0x424c05 <cleanup_source>, arg=0x7f30c77fdca0) at hash.c:400
> >> #6 0x0000000000424ef1 in cleanup_thread (arg=0x0) at cleanup.c:206
> >> #7 0x00007f316b1f9e9a in start_thread (arg=0x7f30c77fe700) at
> >> pthread_create.c:308
> >> #8 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #9 0x0000000000000000 in ?? ()
> >>
> >> Thread 63 (LWP 26982):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e28950) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30c7fff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 62 (LWP 26981):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e270b0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ccff9700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 61 (LWP 26980):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e291b0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cd7fa700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 60 (LWP 26979):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29f30) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cdffb700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 59 (LWP 26978):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29350) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ce7fc700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 58 (LWP 26977):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e27f70) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ceffd700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 57 (LWP 26976):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e27dd0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cf7fe700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 56 (LWP 26975):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2acb0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cffff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 55 (LWP 26974):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2a5f0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3124ff9700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 54 (LWP 26970):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29a10) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31257fa700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 53 (LWP 26969):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e287b0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3125ffb700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 52 (LWP 26968):
> >> #0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
> >> #1 0x00007f316bed4386 in ck_rwlock_write_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:121
> >> #2 0x00007f316bed4d9b in hash_insert (key=0x7f31267fa440,
> >> val=0x7f31267fa450, hash=0x1e23420) at hash.c:210
> >> #3 0x00000000004135e2 in endElement_CLUSTER (data=0x7f31267fa750,
> >> el=0x7f30a800e7f0 "CLUSTER") at process_xml.c:1292
> >> #4 0x00000000004136ba in end (data=0x7f31267fa750, el=0x7f30a800e7f0
> >> "CLUSTER") at process_xml.c:1332
> >> #5 0x00007f316b4196f4 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #6 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #7 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #8 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #9 0x00007f316b41c75d in XML_ParseBuffer () from
> >> /lib/x86_64-linux-gnu/libexpat.so.1
> >> #10 0x00000000004137f2 in process_xml (d=0x1e27590,
> >> buf=0x7f3144778010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
> >> standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
> >> (GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
> >> #REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
> >> #11 0x0000000000409e67 in data_thread (arg=0x1e27590) at
> data_thread.c:287
> >> #12 0x00007f316b1f9e9a in start_thread (arg=0x7f31267fc700) at
> >> pthread_create.c:308
> >> #13 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #14 0x0000000000000000 in ?? ()
> >>
> >> Thread 51 (LWP 26967):
> >> #0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
> >> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:163
> >> #2 0x00007f316bed515f in hash_lookup (key=0x7f3126ffb440,
> hash=0x1e23420)
> >> at hash.c:303
> >> #3 0x0000000000410df7 in startElement_CLUSTER (data=0x7f3126ffb750,
> >> el=0x7f30b51dd7c0 "CLUSTER", attr=0x7f30b52430a0) at process_xml.c:309
> >> #4 0x0000000000412f8d in start (data=0x7f3126ffb750, el=0x7f30b51dd7c0
> >> "CLUSTER", attr=0x7f30b52430a0) at process_xml.c:1106
> >> #5 0x00007f316b419b5c in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #6 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #7 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #8 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #9 0x00007f316b41c75d in XML_ParseBuffer () from
> >> /lib/x86_64-linux-gnu/libexpat.so.1
> >> #10 0x00000000004137f2 in process_xml (d=0x1e26f10,
> >> buf=0x7f309509c010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
> >> standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
> >> (GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
> >> #REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
> >> #11 0x0000000000409e67 in data_thread (arg=0x1e26f10) at
> data_thread.c:287
> >> #12 0x00007f316b1f9e9a in start_thread (arg=0x7f3126ffd700) at
> >> pthread_create.c:308
> >> #13 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >>
> >> #14 0x0000000000000000 in ?? ()
> >>
> >> Thread 50 (LWP 26966):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> ---Type <return> to continue, or q <return> to quit---
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e282d0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31277fe700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 49 (LWP 26965):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e28130) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3127fff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 48 (LWP 26964):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2b6d0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3134ff9700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 47 (LWP 26963):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e28e50) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31357fa700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 46 (LWP 26962):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2b030) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3135ffb700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 45 (LWP 26961):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e28470) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31367fc700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 44 (LWP 26960):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2b530) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3136ffd700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 43 (LWP 26959):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e27c30) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31377fe700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 42 (LWP 26958):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29bd0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3137fff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 41 (LWP 26957):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2b1f0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3144ff9700) at
> >> pthread_create.c:308
> >> ---Type <return> to continue, or q <return> to quit---
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 40 (LWP 26956):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2ba10) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31457fa700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 39 (LWP 26955):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e27a90) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3145ffb700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 38 (LWP 26954):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2bbb0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31467fc700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 37 (LWP 26953):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2a2b0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3146ffd700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 36 (LWP 26952):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e296b0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31477fe700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 35 (LWP 26951):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e24e10) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3147fff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 34 (LWP 26950):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e28cb0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314cff9700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 33 (LWP 26949):
> >> #0 0x00007f316af19f1d in read () at
> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316aeadca8 in _IO_new_file_underflow (fp=0x7f314802f2a0) at
> >> fileops.c:619
> >> #2 0x00007f316aeaecee in _IO_default_uflow (fp=0x7f314802f2a0) at
> >> genops.c:440
> >> #3 0x00007f316aea2e3a in _IO_getline_info (fp=0x7f314802f2a0,
> >> buf=0x7f314d7f2fc0 "0 errors, enqueued 1 value(s).", n=4095, delim=10,
> >> extract_delim=1, eof=0x0) at iogetline.c:74
> >> #4 0x00007f316aea1d1b in _IO_fgets (buf=0x7f314d7f2fc0 "0 errors,
> >> enqueued 1 value(s).", n=<optimized out>, fp=0x7f314802f2a0) at
> iofgets.c:58
> >> #5 0x00007f316bcad64b in fgets (__stream=0x7f314802f2a0, __n=4096,
> >> __s=0x7f314d7f2fc0 "0 errors, enqueued 1 value(s).")
> >> at /usr/include/x86_64-linux-gnu/bits/stdio2.h:255
> >> #6 response_read (ret_response=0x7f314d7f4018) at rrd_client.c:251
> >> #7 0x00007f316bcad860 in request (ret_response=0x7f314d7f4068,
> >> buffer=<optimized out>, buffer_size=<optimized out>) at rrd_client.c:325
> >> #8 request (buffer=<optimized out>, buffer_size=<optimized out>,
> >> ret_response=0x7f314d7f4068) at rrd_client.c:305
> >> #9 0x00007f316bcae1d8 in rrdc_update (filename=<optimized out>,
> >> values_num=1, values=0x7f314d7f71b0) at rrd_client.c:627
> >> #10 0x00007f316bcb42b4 in rrd_update (argc=<optimized out>,
> >> argv=0x7f314d7f71a0) at rrd_update.c:452
> >> #11 0x000000000041627c in RRD_update (rrd=0x7f314d7f7390
> >> "/var/lib/ganglia/rrds/video/
> video13.jfk01.justin.tv/tcp_retrans_percentage.rrd",
> >> sum=0x7f30fa09ae93 "0.07773", num=0x0, process_time=1410382058) at
> >> rrd_helpers.c:212
> >> #12 0x00000000004166f2 in push_data_to_rrd (rrd=0x7f314d7f7390
> >> "/var/lib/ganglia/rrds/video/
> video13.jfk01.justin.tv/tcp_retrans_percentage.rrd",
> >> sum=0x7f30fa09ae93 "0.07773", num=0x0, step=15,
> >> process_time=1410382058, slope=GANGLIA_SLOPE_BOTH) at rrd_helpers.c:344
> >> #13 0x0000000000416b6c in write_data_to_rrd (source=0x7f30f8831660
> >> "video", host=0x7f30f9ff63f0 "video13.jfk01.justin.tv",
> >> metric=0x7f30fa09ae7c "tcp_retrans_percentage", sum=0x7f30fa09ae93
> >> "0.07773", num=0x0, step=15, process_time=1410382058,
> >> slope=GANGLIA_SLOPE_BOTH) at rrd_helpers.c:388
> >> #14 0x0000000000411e86 in startElement_METRIC (data=0x7f314d7f8750,
> >> el=0x7f30f8001b80 "METRIC", attr=0x7f30fa09b290) at process_xml.c:733
> >> #15 0x0000000000412fea in start (data=0x7f314d7f8750, el=0x7f30f8001b80
> >> "METRIC", attr=0x7f30fa09b290) at process_xml.c:1118
> >> #16 0x00007f316b419b5c in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #17 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #18 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #19 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
> >> #20 0x00007f316b41c75d in XML_ParseBuffer () from
> >> /lib/x86_64-linux-gnu/libexpat.so.1
> >> #21 0x00000000004137f2 in process_xml (d=0x1e28610,
> >> buf=0x7f307d77b010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
> >> standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
> >> (GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
> >> #REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
> >> #22 0x0000000000409e67 in data_thread (arg=0x1e28610) at
> data_thread.c:287
> >> #23 0x00007f316b1f9e9a in start_thread (arg=0x7f314d7fa700) at
> >> pthread_create.c:308
> >> #24 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #25 0x0000000000000000 in ?? ()
> >>
> >> Thread 32 (LWP 26948):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2a0f0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314dffb700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 31 (LWP 26947):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e27750) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314e7fc700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 30 (LWP 26946):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2a450) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314effd700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 29 (LWP 26944):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2a790) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314f7fe700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 28 (LWP 26943):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29870) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f314ffff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 27 (LWP 26942):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2b390) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3154ff9700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 26 (LWP 26941):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2aaf0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31557fa700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 25 (LWP 26940):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2a950) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3155ffb700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 24 (LWP 26939):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2b870) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31567fc700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 23 (LWP 26938):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e294f0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3156ffd700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 22 (LWP 26937):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e27250) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f31577fe700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 21 (LWP 26936):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e28b10) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f3157fff700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 20 (LWP 26935):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e273f0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315ca6a700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 19 (LWP 26934):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29d70) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315d26b700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 18 (LWP 26933):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e2ae70) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315da6c700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 17 (LWP 26932):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e29010) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315e26d700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 16 (LWP 26931):
> >> #0 0x00007f316af20683 in select () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
> >> #2 0x0000000000409f33 in data_thread (arg=0x1e278f0) at
> data_thread.c:308
> >> #3 0x00007f316b1f9e9a in start_thread (arg=0x7f315ea6e700) at
> >> pthread_create.c:308
> >> #4 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #5 0x0000000000000000 in ?? ()
> >>
> >> Thread 15 (LWP 26930):
> >> #0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
> >> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:163
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f315f26ec80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f315f26ec80, path=0x0,
> >> myroot=0x7f315f26e440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f315f26ec80,
> >> path=0x7f315f26e470 "/", myroot=0x7f315f26e440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f315f26f700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 14 (LWP 26929):
> >> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> >> /usr/include/gcc/x86_64/ck_pr.h:145
> >> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f315fa6fc80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f315fa6fc80, path=0x0,
> >> myroot=0x7f315fa6f440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f315fa6fc80,
> >> path=0x7f315fa6f470 "/", myroot=0x7f315fa6f440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f315fa70700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 13 (LWP 26928):
> >> #0 0x00007f316bed2436 in ck_pr_stall () at
> >> /usr/include/gcc/x86_64/ck_pr.h:70
> >> #1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:163
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3160270c80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f3160270c80, path=0x0,
> >> myroot=0x7f3160270440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f3160270c80,
> >> path=0x7f3160270470 "/", myroot=0x7f3160270440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3160271700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 12 (LWP 26927):
> >> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> >> /usr/include/gcc/x86_64/ck_pr.h:145
> >> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3160a71c80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f3160a71c80, path=0x0,
> >> myroot=0x7f3160a71440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f3160a71c80,
> >> path=0x7f3160a71470 "/", myroot=0x7f3160a71440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3160a72700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 11 (LWP 26926):
> >> #0 ck_rwlock_read_lock (rw=0x1e23df0) at /usr/include/ck_rwlock.h:163
> >> #1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3161272c80) at hash.c:417
> >> #2 0x000000000040d463 in process_path (client=0x7f3161272c80, path=0x0,
> >> myroot=0x7f3161272440, key=0x0) at server.c:435
> >> #3 0x000000000040d51e in process_path (client=0x7f3161272c80,
> >> path=0x7f3161272470 "/", myroot=0x7f3161272440, key=0x0) at server.c:452
> >> #4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #5 0x00007f316b1f9e9a in start_thread (arg=0x7f3161273700) at
> >> pthread_create.c:308
> >> #6 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #7 0x0000000000000000 in ?? ()
> >>
> >> Thread 10 (LWP 26925):
> >> #0 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3161a73c80) at hash.c:417
> >> #2 0x000000000040d463 in process_path (client=0x7f3161a73c80, path=0x0,
> >> myroot=0x7f3161a73440, key=0x0) at server.c:435
> >> #3 0x000000000040d51e in process_path (client=0x7f3161a73c80,
> >> path=0x7f3161a73470 "/", myroot=0x7f3161a73440, key=0x0) at server.c:452
> >> #4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #5 0x00007f316b1f9e9a in start_thread (arg=0x7f3161a74700) at
> >> pthread_create.c:308
> >> #6 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #7 0x0000000000000000 in ?? ()
> >>
> >> Thread 9 (LWP 26924):
> >> #0 0x00007f316bed44ac in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3162274c80) at hash.c:417
> >> #2 0x000000000040d463 in process_path (client=0x7f3162274c80, path=0x0,
> >> myroot=0x7f3162274440, key=0x0) at server.c:435
> >> #3 0x000000000040d51e in process_path (client=0x7f3162274c80,
> >> path=0x7f3162274470 "/", myroot=0x7f3162274440, key=0x0) at server.c:452
> >> #4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #5 0x00007f316b1f9e9a in start_thread (arg=0x7f3162275700) at
> >> pthread_create.c:308
> >> #6 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #7 0x0000000000000000 in ?? ()
> >>
> >> Thread 8 (LWP 26923):
> >> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> >> /usr/include/gcc/x86_64/ck_pr.h:145
> >> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3162a75c80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f3162a75c80, path=0x0,
> >> myroot=0x7f3162a75440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f3162a75c80,
> >> path=0x7f3162a75470 "/", myroot=0x7f3162a75440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3162a76700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 7 (LWP 26922):
> >> #0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
> >> /usr/include/gcc/x86_64/ck_pr.h:145
> >> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3163276c80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f3163276c80, path=0x0,
> >> myroot=0x7f3163276440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f3163276c80,
> >> path=0x7f3163276470 "/", myroot=0x7f3163276440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3163277700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 6 (LWP 26921):
> >> #0 0x00007f316bed2631 in ck_pr_load_uint (target=0x1e23df0) at
> >> /usr/include/gcc/x86_64/ck_pr.h:145
> >> #1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
> >> /usr/include/ck_rwlock.h:162
> >> #2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
> >> <tree_report>, arg=0x7f3163a77c80) at hash.c:417
> >> #3 0x000000000040d463 in process_path (client=0x7f3163a77c80, path=0x0,
> >> myroot=0x7f3163a77440, key=0x0) at server.c:435
> >> #4 0x000000000040d51e in process_path (client=0x7f3163a77c80,
> >> path=0x7f3163a77470 "/", myroot=0x7f3163a77440, key=0x0) at server.c:452
> >> #5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
> >> #6 0x00007f316b1f9e9a in start_thread (arg=0x7f3163a78700) at
> >> pthread_create.c:308
> >> #7 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #8 0x0000000000000000 in ?? ()
> >>
> >> Thread 5 (LWP 26920):
> >> #0 __lll_lock_wait () at
> >> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> >> #1 0x00007f316b1fc065 in _L_lock_858 () from
> >> /lib/x86_64-linux-gnu/libpthread.so.0
> >> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> >> pthread_mutex_lock.c:61
> >> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> >> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f3164279700) at
> >> pthread_create.c:308
> >> #5 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #6 0x0000000000000000 in ?? ()
> >>
> >> Thread 4 (LWP 26919):
> >> #0 0x00007f316b20104d in accept () at
> >> ../sysdeps/unix/syscall-template.S:82
> >> #1 0x000000000040dd64 in server_thread (arg=0x0) at server.c:724
> >> #2 0x00007f316b1f9e9a in start_thread (arg=0x7f3164a7a700) at
> >> pthread_create.c:308
> >> #3 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #4 0x0000000000000000 in ?? ()
> >>
> >> Thread 3 (LWP 26918):
> >> #0 __lll_lock_wait () at
> >> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> >> #1 0x00007f316b1fc065 in _L_lock_858 () from
> >> /lib/x86_64-linux-gnu/libpthread.so.0
> >> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> >> pthread_mutex_lock.c:61
> >> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> >> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f316527b700) at
> >> pthread_create.c:308
> >> #5 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #6 0x0000000000000000 in ?? ()
> >>
> >> Thread 2 (LWP 26917):
> >> #0 __lll_lock_wait () at
> >> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> >> #1 0x00007f316b1fc065 in _L_lock_858 () from
> >> /lib/x86_64-linux-gnu/libpthread.so.0
> >> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> >> pthread_mutex_lock.c:61
> >> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> >> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f3165a7c700) at
> >> pthread_create.c:308
> >> #5 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #6 0x0000000000000000 in ?? ()
> >>
> >> Thread 1 (LWP 26916):
> >> #0 __lll_lock_wait () at
> >> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
> >> #1 0x00007f316b1fc065 in _L_lock_858 () from
> >> /lib/x86_64-linux-gnu/libpthread.so.0
> >> #2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
> >> pthread_mutex_lock.c:61
> >> #3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
> >> #4 0x00007f316b1f9e9a in start_thread (arg=0x7f316627d700) at
> >> pthread_create.c:308
> >> #5 0x00007f316af2731d in clone () at
> >> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> >> #6 0x0000000000000000 in ?? ()
> >> (gdb)
> >>
> >> (gdb) quit
> >>
> >>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Want excitement?
> >> Manually upgrade your production database.
> >> When you want reliability, choose Perforce
> >> Perforce version control. Predictably reliable.
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> >>
> >>
> >>
> >> _______________________________________________
> >> Ganglia-developers mailing list
> >> Ganglia-developers@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/ganglia-developers
> >>
> >>
> >>
> >>
> >>
> ------------------------------------------------------------------------------
> >> Want excitement?
> >> Manually upgrade your production database.
> >> When you want reliability, choose Perforce
> >> Perforce version control. Predictably reliable.
> >>
> >>
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> >> _______________________________________________
> >> Ganglia-developers mailing list
> >> Ganglia-developers@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/ganglia-developers
> >>
> >
> >
> >
> ------------------------------------------------------------------------------
> > Want excitement?
> > Manually upgrade your production database.
> > When you want reliability, choose Perforce
> > Perforce version control. Predictably reliable.
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Ganglia-developers mailing list
> > Ganglia-developers@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/ganglia-developers
> >
>
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
