We're seeing gmetad (version 3.7.0) threads segfaulting or gmetad and all
its threads spinning 100% cpu. The system is running ubuntu 12.04/precise.
Our ganglia has 48 data source, 1300 hosts and has around 220k metrics. We
are running this with rrdcached. Would like to fix it from segfault and
spin on cpu. I'm including a backtrace of when gmetad and threads were
spinning on cpu.
Is there anything I can provide to help figure this out?
gmetad.conf looks like this:
RRAs "RRA:AVERAGE:0.5:1:5856" "RRA:AVERAGE:0.5:24:1680"
"RRA:AVERAGE:0.5:168:244" "RRA:AVERAGE:0.5:672:274"
"RRA:AVERAGE:0.5:5760:3740"
server_threads 10
case_sensitive_hostnames 1
and includes 48 data_source tags and each has 2 data sources.
ganglia was ./configured using:
./configure --host=x86_64-linux-gnu --build=x86_64-linux-gnu --prefix=/usr
--mandir=${prefix}/share/man --libdir=${prefix}/lib
--sysconfdir=/etc/ganglia --infodir=${prefix}/share/info --enable-shared
--with-gmetad --with-memcached --enable-debug
Backtrace:
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/sbin/gmetad...done.
[New LWP 26916]
[New LWP 26917]
[New LWP 26918]
[New LWP 26919]
[New LWP 26920]
[New LWP 26921]
[New LWP 26922]
[New LWP 26923]
[New LWP 26924]
[New LWP 26925]
[New LWP 26926]
[New LWP 26927]
[New LWP 26928]
[New LWP 26929]
[New LWP 26930]
[New LWP 26931]
[New LWP 26932]
[New LWP 26933]
[New LWP 26934]
[New LWP 26935]
[New LWP 26936]
[New LWP 26937]
[New LWP 26938]
[New LWP 26939]
[New LWP 26940]
[New LWP 26941]
[New LWP 26942]
[New LWP 26943]
[New LWP 26944]
[New LWP 26946]
[New LWP 26947]
[New LWP 26948]
[New LWP 26949]
[New LWP 26950]
[New LWP 26951]
[New LWP 26952]
[New LWP 26953]
[New LWP 26954]
[New LWP 26955]
[New LWP 26956]
[New LWP 26957]
[New LWP 26958]
[New LWP 26959]
[New LWP 26960]
[New LWP 26961]
[New LWP 26962]
[New LWP 26963]
[New LWP 26964]
[New LWP 26965]
[New LWP 26966]
[New LWP 26967]
[New LWP 26968]
[New LWP 26969]
[New LWP 26970]
[New LWP 26974]
[New LWP 26975]
[New LWP 26976]
[New LWP 26977]
[New LWP 26978]
[New LWP 26979]
[New LWP 26980]
[New LWP 26981]
[New LWP 26982]
[New LWP 26983]
[New LWP 26908]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `gmetad'.
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
132 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file
or directory.
(gdb) thread apply all bt
Thread 65 (LWP 26908):
#0 0x00007f316bed2436 in ck_pr_stall () at
/usr/include/gcc/x86_64/ck_pr.h:70
#1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:163
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x4057d1
<do_root_summary>, arg=0x0) at hash.c:417
#3 0x00000000004064f5 in main (argc=2, argv=0x7fff547ded18) at gmetad.c:586
Thread 64 (LWP 26983):
#0 0x00007f316bed2533 in ck_pr_fas_uint (target=0x7f30b402c230, v=1) at
/usr/include/gcc/x86_64/ck_pr.h:115
#1 0x00007f316bed4376 in ck_rwlock_write_lock (rw=0x7f30b402c230) at
/usr/include/ck_rwlock.h:115
#2 0x00007f316bed528f in hash_delete (key=0x7f30b48792e0,
hash=0x7f30b40f4240) at hash.c:335
#3 0x00007f316bed4a75 in hash_destroy (hash=0x7f30b40f4240) at hash.c:127
#4 0x0000000000424d47 in cleanup_source (key=0x7f30b49dd210,
val=0x7f30b49dd140, arg=0x7f30c77fdca0) at cleanup.c:170
#5 0x00007f316bed54c0 in hash_walkfrom (hash=0x1e23420, from=0,
func=0x424c05 <cleanup_source>, arg=0x7f30c77fdca0) at hash.c:400
#6 0x0000000000424ef1 in cleanup_thread (arg=0x0) at cleanup.c:206
#7 0x00007f316b1f9e9a in start_thread (arg=0x7f30c77fe700) at
pthread_create.c:308
#8 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9 0x0000000000000000 in ?? ()
Thread 63 (LWP 26982):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e28950) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30c7fff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 62 (LWP 26981):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e270b0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ccff9700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 61 (LWP 26980):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e291b0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cd7fa700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 60 (LWP 26979):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29f30) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cdffb700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 59 (LWP 26978):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29350) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ce7fc700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 58 (LWP 26977):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e27f70) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30ceffd700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 57 (LWP 26976):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e27dd0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cf7fe700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 56 (LWP 26975):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2acb0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f30cffff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 55 (LWP 26974):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2a5f0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3124ff9700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 54 (LWP 26970):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29a10) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31257fa700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 53 (LWP 26969):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e287b0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3125ffb700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 52 (LWP 26968):
#0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
#1 0x00007f316bed4386 in ck_rwlock_write_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:121
#2 0x00007f316bed4d9b in hash_insert (key=0x7f31267fa440,
val=0x7f31267fa450, hash=0x1e23420) at hash.c:210
#3 0x00000000004135e2 in endElement_CLUSTER (data=0x7f31267fa750,
el=0x7f30a800e7f0 "CLUSTER") at process_xml.c:1292
#4 0x00000000004136ba in end (data=0x7f31267fa750, el=0x7f30a800e7f0
"CLUSTER") at process_xml.c:1332
#5 0x00007f316b4196f4 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#6 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#7 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#8 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#9 0x00007f316b41c75d in XML_ParseBuffer () from
/lib/x86_64-linux-gnu/libexpat.so.1
#10 0x00000000004137f2 in process_xml (d=0x1e27590,
buf=0x7f3144778010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
(GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
#REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
#11 0x0000000000409e67 in data_thread (arg=0x1e27590) at data_thread.c:287
#12 0x00007f316b1f9e9a in start_thread (arg=0x7f31267fc700) at
pthread_create.c:308
#13 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()
Thread 51 (LWP 26967):
#0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
#1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:163
#2 0x00007f316bed515f in hash_lookup (key=0x7f3126ffb440, hash=0x1e23420)
at hash.c:303
#3 0x0000000000410df7 in startElement_CLUSTER (data=0x7f3126ffb750,
el=0x7f30b51dd7c0 "CLUSTER", attr=0x7f30b52430a0) at process_xml.c:309
#4 0x0000000000412f8d in start (data=0x7f3126ffb750, el=0x7f30b51dd7c0
"CLUSTER", attr=0x7f30b52430a0) at process_xml.c:1106
#5 0x00007f316b419b5c in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#6 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#7 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#8 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#9 0x00007f316b41c75d in XML_ParseBuffer () from
/lib/x86_64-linux-gnu/libexpat.so.1
#10 0x00000000004137f2 in process_xml (d=0x1e26f10,
buf=0x7f309509c010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
(GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
#REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
#11 0x0000000000409e67 in data_thread (arg=0x1e26f10) at data_thread.c:287
#12 0x00007f316b1f9e9a in start_thread (arg=0x7f3126ffd700) at
pthread_create.c:308
#13 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()
Thread 50 (LWP 26966):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
---Type <return> to continue, or q <return> to quit---
#2 0x0000000000409f33 in data_thread (arg=0x1e282d0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31277fe700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 49 (LWP 26965):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e28130) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3127fff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 48 (LWP 26964):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2b6d0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3134ff9700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 47 (LWP 26963):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e28e50) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31357fa700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 46 (LWP 26962):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2b030) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3135ffb700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 45 (LWP 26961):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e28470) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31367fc700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 44 (LWP 26960):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2b530) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3136ffd700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 43 (LWP 26959):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e27c30) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31377fe700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 42 (LWP 26958):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29bd0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3137fff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 41 (LWP 26957):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2b1f0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3144ff9700) at
pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 40 (LWP 26956):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2ba10) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31457fa700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 39 (LWP 26955):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e27a90) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3145ffb700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 38 (LWP 26954):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2bbb0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31467fc700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 37 (LWP 26953):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2a2b0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3146ffd700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 36 (LWP 26952):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e296b0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31477fe700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 35 (LWP 26951):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e24e10) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3147fff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 34 (LWP 26950):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e28cb0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f314cff9700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 33 (LWP 26949):
#0 0x00007f316af19f1d in read () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316aeadca8 in _IO_new_file_underflow (fp=0x7f314802f2a0) at
fileops.c:619
#2 0x00007f316aeaecee in _IO_default_uflow (fp=0x7f314802f2a0) at
genops.c:440
#3 0x00007f316aea2e3a in _IO_getline_info (fp=0x7f314802f2a0,
buf=0x7f314d7f2fc0 "0 errors, enqueued 1 value(s).", n=4095, delim=10,
extract_delim=1, eof=0x0) at iogetline.c:74
#4 0x00007f316aea1d1b in _IO_fgets (buf=0x7f314d7f2fc0 "0 errors, enqueued
1 value(s).", n=<optimized out>, fp=0x7f314802f2a0) at iofgets.c:58
#5 0x00007f316bcad64b in fgets (__stream=0x7f314802f2a0, __n=4096,
__s=0x7f314d7f2fc0 "0 errors, enqueued 1 value(s).")
at /usr/include/x86_64-linux-gnu/bits/stdio2.h:255
#6 response_read (ret_response=0x7f314d7f4018) at rrd_client.c:251
#7 0x00007f316bcad860 in request (ret_response=0x7f314d7f4068,
buffer=<optimized out>, buffer_size=<optimized out>) at rrd_client.c:325
#8 request (buffer=<optimized out>, buffer_size=<optimized out>,
ret_response=0x7f314d7f4068) at rrd_client.c:305
#9 0x00007f316bcae1d8 in rrdc_update (filename=<optimized out>,
values_num=1, values=0x7f314d7f71b0) at rrd_client.c:627
#10 0x00007f316bcb42b4 in rrd_update (argc=<optimized out>,
argv=0x7f314d7f71a0) at rrd_update.c:452
#11 0x000000000041627c in RRD_update (rrd=0x7f314d7f7390
"/var/lib/ganglia/rrds/video/
video13.jfk01.justin.tv/tcp_retrans_percentage.rrd",
sum=0x7f30fa09ae93 "0.07773", num=0x0, process_time=1410382058) at
rrd_helpers.c:212
#12 0x00000000004166f2 in push_data_to_rrd (rrd=0x7f314d7f7390
"/var/lib/ganglia/rrds/video/
video13.jfk01.justin.tv/tcp_retrans_percentage.rrd",
sum=0x7f30fa09ae93 "0.07773", num=0x0, step=15,
process_time=1410382058, slope=GANGLIA_SLOPE_BOTH) at rrd_helpers.c:344
#13 0x0000000000416b6c in write_data_to_rrd (source=0x7f30f8831660 "video",
host=0x7f30f9ff63f0 "video13.jfk01.justin.tv",
metric=0x7f30fa09ae7c "tcp_retrans_percentage", sum=0x7f30fa09ae93
"0.07773", num=0x0, step=15, process_time=1410382058,
slope=GANGLIA_SLOPE_BOTH) at rrd_helpers.c:388
#14 0x0000000000411e86 in startElement_METRIC (data=0x7f314d7f8750,
el=0x7f30f8001b80 "METRIC", attr=0x7f30fa09b290) at process_xml.c:733
#15 0x0000000000412fea in start (data=0x7f314d7f8750, el=0x7f30f8001b80
"METRIC", attr=0x7f30fa09b290) at process_xml.c:1118
#16 0x00007f316b419b5c in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#17 0x00007f316b41a951 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#18 0x00007f316b4177c7 in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#19 0x00007f316b41917b in ?? () from /lib/x86_64-linux-gnu/libexpat.so.1
#20 0x00007f316b41c75d in XML_ParseBuffer () from
/lib/x86_64-linux-gnu/libexpat.so.1
#21 0x00000000004137f2 in process_xml (d=0x1e28610,
buf=0x7f307d77b010 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"
standalone=\"yes\"?>\n<!DOCTYPE GANGLIA_XML [\n <!ELEMENT GANGLIA_XML
(GRID|CLUSTER|HOST)*>\n <!ATTLIST GANGLIA_XML VERSION CDATA
#REQUIRED>\n <!ATTLIST"...) at process_xml.c:1376
#22 0x0000000000409e67 in data_thread (arg=0x1e28610) at data_thread.c:287
#23 0x00007f316b1f9e9a in start_thread (arg=0x7f314d7fa700) at
pthread_create.c:308
#24 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#25 0x0000000000000000 in ?? ()
Thread 32 (LWP 26948):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2a0f0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f314dffb700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 31 (LWP 26947):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e27750) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f314e7fc700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 30 (LWP 26946):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2a450) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f314effd700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 29 (LWP 26944):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2a790) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f314f7fe700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 28 (LWP 26943):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29870) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f314ffff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 27 (LWP 26942):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2b390) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3154ff9700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 26 (LWP 26941):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2aaf0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31557fa700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 25 (LWP 26940):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2a950) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3155ffb700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 24 (LWP 26939):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2b870) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31567fc700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 23 (LWP 26938):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e294f0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3156ffd700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 22 (LWP 26937):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e27250) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f31577fe700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 21 (LWP 26936):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e28b10) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f3157fff700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 20 (LWP 26935):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e273f0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f315ca6a700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 19 (LWP 26934):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29d70) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f315d26b700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 18 (LWP 26933):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e2ae70) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f315da6c700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 17 (LWP 26932):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e29010) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f315e26d700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 16 (LWP 26931):
#0 0x00007f316af20683 in select () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f316c1170fd in apr_sleep () from /usr/lib/libapr-1.so.0
#2 0x0000000000409f33 in data_thread (arg=0x1e278f0) at data_thread.c:308
#3 0x00007f316b1f9e9a in start_thread (arg=0x7f315ea6e700) at
pthread_create.c:308
#4 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
Thread 15 (LWP 26930):
#0 ck_pr_stall () at /usr/include/gcc/x86_64/ck_pr.h:70
#1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:163
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f315f26ec80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f315f26ec80, path=0x0,
myroot=0x7f315f26e440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f315f26ec80,
path=0x7f315f26e470 "/", myroot=0x7f315f26e440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f315f26f700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 14 (LWP 26929):
#0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
/usr/include/gcc/x86_64/ck_pr.h:145
#1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f315fa6fc80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f315fa6fc80, path=0x0,
myroot=0x7f315fa6f440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f315fa6fc80,
path=0x7f315fa6f470 "/", myroot=0x7f315fa6f440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f315fa70700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 13 (LWP 26928):
#0 0x00007f316bed2436 in ck_pr_stall () at
/usr/include/gcc/x86_64/ck_pr.h:70
#1 0x00007f316bed44a8 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:163
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3160270c80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f3160270c80, path=0x0,
myroot=0x7f3160270440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f3160270c80,
path=0x7f3160270470 "/", myroot=0x7f3160270440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f3160271700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 12 (LWP 26927):
#0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
/usr/include/gcc/x86_64/ck_pr.h:145
#1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3160a71c80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f3160a71c80, path=0x0,
myroot=0x7f3160a71440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f3160a71c80,
path=0x7f3160a71470 "/", myroot=0x7f3160a71440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f3160a72700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 11 (LWP 26926):
#0 ck_rwlock_read_lock (rw=0x1e23df0) at /usr/include/ck_rwlock.h:163
#1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3161272c80) at hash.c:417
#2 0x000000000040d463 in process_path (client=0x7f3161272c80, path=0x0,
myroot=0x7f3161272440, key=0x0) at server.c:435
#3 0x000000000040d51e in process_path (client=0x7f3161272c80,
path=0x7f3161272470 "/", myroot=0x7f3161272440, key=0x0) at server.c:452
#4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#5 0x00007f316b1f9e9a in start_thread (arg=0x7f3161273700) at
pthread_create.c:308
#6 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
Thread 10 (LWP 26925):
#0 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3161a73c80) at hash.c:417
#2 0x000000000040d463 in process_path (client=0x7f3161a73c80, path=0x0,
myroot=0x7f3161a73440, key=0x0) at server.c:435
#3 0x000000000040d51e in process_path (client=0x7f3161a73c80,
path=0x7f3161a73470 "/", myroot=0x7f3161a73440, key=0x0) at server.c:452
#4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#5 0x00007f316b1f9e9a in start_thread (arg=0x7f3161a74700) at
pthread_create.c:308
#6 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
Thread 9 (LWP 26924):
#0 0x00007f316bed44ac in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#1 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3162274c80) at hash.c:417
#2 0x000000000040d463 in process_path (client=0x7f3162274c80, path=0x0,
myroot=0x7f3162274440, key=0x0) at server.c:435
#3 0x000000000040d51e in process_path (client=0x7f3162274c80,
path=0x7f3162274470 "/", myroot=0x7f3162274440, key=0x0) at server.c:452
#4 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#5 0x00007f316b1f9e9a in start_thread (arg=0x7f3162275700) at
pthread_create.c:308
#6 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
Thread 8 (LWP 26923):
#0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
/usr/include/gcc/x86_64/ck_pr.h:145
#1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3162a75c80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f3162a75c80, path=0x0,
myroot=0x7f3162a75440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f3162a75c80,
path=0x7f3162a75470 "/", myroot=0x7f3162a75440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f3162a76700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 7 (LWP 26922):
#0 0x00007f316bed262e in ck_pr_load_uint (target=0x1e23df0) at
/usr/include/gcc/x86_64/ck_pr.h:145
#1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3163276c80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f3163276c80, path=0x0,
myroot=0x7f3163276440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f3163276c80,
path=0x7f3163276470 "/", myroot=0x7f3163276440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f3163277700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 6 (LWP 26921):
#0 0x00007f316bed2631 in ck_pr_load_uint (target=0x1e23df0) at
/usr/include/gcc/x86_64/ck_pr.h:145
#1 0x00007f316bed44b4 in ck_rwlock_read_lock (rw=0x1e23df0) at
/usr/include/ck_rwlock.h:162
#2 0x00007f316bed556c in hash_foreach (hash=0x1e23420, func=0x40d313
<tree_report>, arg=0x7f3163a77c80) at hash.c:417
#3 0x000000000040d463 in process_path (client=0x7f3163a77c80, path=0x0,
myroot=0x7f3163a77440, key=0x0) at server.c:435
#4 0x000000000040d51e in process_path (client=0x7f3163a77c80,
path=0x7f3163a77470 "/", myroot=0x7f3163a77440, key=0x0) at server.c:452
#5 0x000000000040e009 in server_thread (arg=0x1) at server.c:796
#6 0x00007f316b1f9e9a in start_thread (arg=0x7f3163a78700) at
pthread_create.c:308
#7 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8 0x0000000000000000 in ?? ()
Thread 5 (LWP 26920):
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1 0x00007f316b1fc065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
pthread_mutex_lock.c:61
#3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
#4 0x00007f316b1f9e9a in start_thread (arg=0x7f3164279700) at
pthread_create.c:308
#5 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 4 (LWP 26919):
#0 0x00007f316b20104d in accept () at ../sysdeps/unix/syscall-template.S:82
#1 0x000000000040dd64 in server_thread (arg=0x0) at server.c:724
#2 0x00007f316b1f9e9a in start_thread (arg=0x7f3164a7a700) at
pthread_create.c:308
#3 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4 0x0000000000000000 in ?? ()
Thread 3 (LWP 26918):
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1 0x00007f316b1fc065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
pthread_mutex_lock.c:61
#3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
#4 0x00007f316b1f9e9a in start_thread (arg=0x7f316527b700) at
pthread_create.c:308
#5 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 2 (LWP 26917):
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1 0x00007f316b1fc065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
pthread_mutex_lock.c:61
#3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
#4 0x00007f316b1f9e9a in start_thread (arg=0x7f3165a7c700) at
pthread_create.c:308
#5 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
Thread 1 (LWP 26916):
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1 0x00007f316b1fc065 in _L_lock_858 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f316b1fbeba in __pthread_mutex_lock (mutex=0x644200) at
pthread_mutex_lock.c:61
#3 0x000000000040dd3f in server_thread (arg=0x0) at server.c:723
#4 0x00007f316b1f9e9a in start_thread (arg=0x7f316627d700) at
pthread_create.c:308
#5 0x00007f316af2731d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6 0x0000000000000000 in ?? ()
(gdb)
(gdb) quit
------------------------------------------------------------------------------
Want excitement?
Manually upgrade your production database.
When you want reliability, choose Perforce
Perforce version control. Predictably reliable.
http://pubads.g.doubleclick.net/gampad/clk?id=157508191&iu=/4140/ostg.clktrk
_______________________________________________
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
