(Sorry, once more with actual attachments.)
Kern Sibbald wrote:
> At this point, before sending anything, first, please ensure the patch is
> applied. If so, 90% probability you will not have any more problems. If you
> do, the lock manager will produce a nice dump with additional information --
> if it is not emailed to you, you should find two files in your working
> directory that contain the traceback and the bactrace outputs.
>
> The lock manager will not prevent lockups, but it will detect deadlock
> situations, and then blow up the SD so as to produce a useful dump.
I ran into this problem again last night (symptoms: no response after
"Used volume status:" when running "status storage" on bconsole; extra
bacula-sd process, which strace shows is running futex() over and over
again), and managed to get the traceback and the lock dump.
Unfortunately, the deadlock detection did not seem to work; I left
things hung for about 20 minutes or so before running "kill -6" on the
parent SD process. (That still left the child, so I had to "kill -9"
that one.) However, I'm hoping that the info is still useful; if so,
let me know and I'll file a bug.
And now for some uninformed speculation:
Looking at the backtrace and the lock dump, it seems that one thread
(0x4519d940) held the two locks that were being waited for by other
threads. In turn, that lock-holding thread had finished a job (jcr
0x12a92d58), ran release_device (dcr 0x12ac15b8), and was running the
alert command (which is set to "sh -c '/usr/sbin/smartctl -H -l error
-q errorsonly -d scsi %c'") and waiting for output. I'm assuming that
the thread was still waiting for this when I killed it.
Looking at the code for release_device(), it seems that the alert
command is called without the optional watchdog timer:
alert = get_pool_memory(PM_FNAME);
alert = edit_device_codes(dcr, alert, dcr->device->alert_command, "");
bpipe = open_bpipe(alert, 0, "r");
(Line 529-531 in acquire.c, version 5.0.1)
Obviously, if something's wrong w/the alert command or my hardware,
that's bad. But would it be a good thing to call the alert command
with, say, a 60-second watchdog timer to avoid this kind of problem?
If there are other issues at work that make this just a workaround,
wouldn't it still be good to be alerted that there's a problem? (Here
I'm assuming that either the lock manager could do this (and
kill/segfault the process, producing a backtrace), or that the timeout
could be caught w/o greater harm and turned into a log message.)
Natch, I'm not a programmer (let alone a Bacula dev), nor do I play
one on TV, but I'm very curious about what's going on under the hood.
If I've mistaken something or missed the point entirely, I'd be
grateful if someone could point it out.
Thanks again for your time!
--
Hugh Brown, Systems Manager
The Centre for High-Throughput Biology
[email protected]
[Thread debugging using libthread_db enabled]
[New Thread 0x2ba33b978560 (LWP 9518)]
[New Thread 0x4d3aa940 (LWP 2845)]
[New Thread 0x4ddab940 (LWP 2840)]
[New Thread 0x483a2940 (LWP 2811)]
[New Thread 0x479a1940 (LWP 2809)]
[New Thread 0x46fa0940 (LWP 2808)]
[New Thread 0x4659f940 (LWP 2807)]
[New Thread 0x45b9e940 (LWP 2806)]
[New Thread 0x4519d940 (LWP 2805)]
[New Thread 0x4339a940 (LWP 2804)]
[New Thread 0x41960940 (LWP 2803)]
[New Thread 0x43d9b940 (LWP 2802)]
[New Thread 0x42999940 (LWP 9523)]
0x00000035ab00e4af in waitpid () from /lib64/libpthread.so.0
$1 = "cbs-01-sd", '\0' <repeats 20 times>
$2 = 0x12a35528 "bacula-sd"
$3 = 0x12a35568 "/usr/sbin/bacula-sd"
$4 = 0x0
$5 = 0x31b0a450e6 "5.0.1 (24 February 2010)"
$6 = 0x31b0a450ff "x86_64-redhat-linux-gnu"
$7 = 0x31b0a45117 "redhat"
$8 = 0x31b0a44ddd ""
$9 = "agnatha.chibi.ubc.ca", '\0' <repeats 29 times>
#0 0x00000035ab00e4af in waitpid () from /lib64/libpthread.so.0
#1 0x00000031b0a31fd0 in signal_handler (sig=<value optimized out>) at
signal.c:229
#2 <signal handler called>
#3 0x00000035aa4cced2 in select () from /lib64/libc.so.6
#4 0x00000031b0a1603b in bnet_thread_server (addrs=<value optimized out>,
max_clients=<value optimized out>, client_wq=<value optimized out>,
handle_client_request=<value optimized out>) at bnet_server.c:161
#5 0x000000000040796d in main (argc=<value optimized out>, argv=<value
optimized out>) at stored.c:312
Thread 13 (Thread 0x42999940 (LWP 9523)):
#0 0x00000035ab00af70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a3fa01 in bthread_cond_timedwait_p (cond=<value optimized out>,
m=<value optimized out>, abstime=<value optimized out>, file=<value optimized
out>,
line=<value optimized out>) at lockmgr.c:786
#2 0x00000031b0a3ad1a in watchdog_thread (arg=<value optimized out>) at
watchdog.c:308
#3 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#4 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#5 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x43d9b940 (LWP 2802)):
#0 0x00000035ab00d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00000035ab008c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x00000035ab008afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000031b0a3fc7b in lmgr_p (m=<value optimized out>) at lockmgr.c:95
#4 0x00000031b0a3fded in bthread_mutex_lock_p (m=<value optimized out>,
file=<value optimized out>, line=<value optimized out>) at lockmgr.c:718
#5 0x00000000004277f9 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:222
#6 0x000000000042796b in DEVICE::dblock (this=0x12a68dc0, why=7) at lock.c:145
#7 0x00000000004336c4 in despool_data (dcr=0x12a923f8, commit=true) at
dev.h:557
#8 0x0000000000434062 in commit_data_spool (dcr=0x12a923f8) at spool.c:136
#9 0x000000000040f829 in do_append_data (jcr=0x12a56ab8) at append.c:318
#10 0x0000000000422def in append_data_cmd (jcr=0x12a56ab8) at fd_cmds.c:203
#11 0x00000000004225a3 in do_fd_commands (jcr=0x12a56ab8) at fd_cmds.c:162
#12 0x0000000000422d09 in run_job (jcr=0x12a56ab8) at fd_cmds.c:124
#13 0x0000000000423383 in run_cmd (jcr=0x12a56ab8) at job.c:225
#14 0x000000000041fbe8 in handle_connection_request (arg=0x12aa7008) at
dircmd.c:233
#15 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#16 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#17 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#18 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x41960940 (LWP 2803)):
#0 0x00000035ab00ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a3f4f6 in bthread_cond_wait_p (cond=<value optimized out>,
m=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
at lockmgr.c:801
#2 0x0000000000427860 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:233
#3 0x000000000042796b in DEVICE::dblock (this=0x12a68e54, why=7) at lock.c:145
#4 0x00000000004336c4 in despool_data (dcr=0x12a928a8, commit=true) at
dev.h:557
#5 0x0000000000434062 in commit_data_spool (dcr=0x12a928a8) at spool.c:136
#6 0x000000000040f829 in do_append_data (jcr=0x12aa75e8) at append.c:318
#7 0x0000000000422def in append_data_cmd (jcr=0x12aa75e8) at fd_cmds.c:203
#8 0x00000000004225a3 in do_fd_commands (jcr=0x12aa75e8) at fd_cmds.c:162
#9 0x0000000000422d09 in run_job (jcr=0x12aa75e8) at fd_cmds.c:124
#10 0x0000000000423383 in run_cmd (jcr=0x12aa75e8) at job.c:225
#11 0x000000000041fbe8 in handle_connection_request (arg=0x12a92108) at
dircmd.c:233
#12 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#13 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#14 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#15 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x4339a940 (LWP 2804)):
#0 0x00000035ab00d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00000035ab008c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x00000035ab008afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000031b0a3fc7b in lmgr_p (m=<value optimized out>) at lockmgr.c:95
#4 0x00000031b0a3fded in bthread_mutex_lock_p (m=<value optimized out>,
file=<value optimized out>, line=<value optimized out>) at lockmgr.c:718
#5 0x00000000004277f9 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:222
#6 0x000000000042796b in DEVICE::dblock (this=0x12a68dc0, why=7) at lock.c:145
#7 0x00000000004336c4 in despool_data (dcr=0x12a91008, commit=true) at
dev.h:557
#8 0x0000000000434062 in commit_data_spool (dcr=0x12a91008) at spool.c:136
#9 0x000000000040f829 in do_append_data (jcr=0x12a6fb88) at append.c:318
#10 0x0000000000422def in append_data_cmd (jcr=0x12a6fb88) at fd_cmds.c:203
#11 0x00000000004225a3 in do_fd_commands (jcr=0x12a6fb88) at fd_cmds.c:162
#12 0x0000000000422d09 in run_job (jcr=0x12a6fb88) at fd_cmds.c:124
#13 0x0000000000423383 in run_cmd (jcr=0x12a6fb88) at job.c:225
#14 0x000000000041fbe8 in handle_connection_request (arg=0x12aa7a88) at
dircmd.c:233
#15 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#16 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#17 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#18 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x4519d940 (LWP 2805)):
#0 0x00000035aa4c5f3b in read () from /lib64/libc.so.6
#1 0x00000035aa46cc07 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00000035aa46d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00000035aa468e8b in getc () from /lib64/libc.so.6
#4 0x00000031b0a135c0 in bfgets (s=<value optimized out>, size=<value
optimized out>, fd=<value optimized out>) at bsys.c:617
#5 0x000000000040d432 in release_device (dcr=0x12ac15b8) at acquire.c:533
#6 0x000000000040f83b in do_append_data (jcr=0x12a92d58) at append.c:329
#7 0x0000000000422def in append_data_cmd (jcr=0x12a92d58) at fd_cmds.c:203
#8 0x00000000004225a3 in do_fd_commands (jcr=0x12a92d58) at fd_cmds.c:162
#9 0x0000000000422d09 in run_job (jcr=0x12a92d58) at fd_cmds.c:124
#10 0x0000000000423383 in run_cmd (jcr=0x12a92d58) at job.c:225
#11 0x000000000041fbe8 in handle_connection_request (arg=0x12aa0b28) at
dircmd.c:233
#12 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#13 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#14 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#15 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x45b9e940 (LWP 2806)):
#0 0x00000035ab00ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a3f4f6 in bthread_cond_wait_p (cond=<value optimized out>,
m=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
at lockmgr.c:801
#2 0x0000000000427860 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:233
#3 0x000000000042796b in DEVICE::dblock (this=0x12a68e54, why=7) at lock.c:145
#4 0x00000000004336c4 in despool_data (dcr=0x12ac1af8, commit=true) at
dev.h:557
#5 0x0000000000434062 in commit_data_spool (dcr=0x12ac1af8) at spool.c:136
#6 0x000000000040f829 in do_append_data (jcr=0x12a946f8) at append.c:318
#7 0x0000000000422def in append_data_cmd (jcr=0x12a946f8) at fd_cmds.c:203
#8 0x00000000004225a3 in do_fd_commands (jcr=0x12a946f8) at fd_cmds.c:162
#9 0x0000000000422d09 in run_job (jcr=0x12a946f8) at fd_cmds.c:124
#10 0x0000000000423383 in run_cmd (jcr=0x12a946f8) at job.c:225
#11 0x000000000041fbe8 in handle_connection_request (arg=0x12aa0d78) at
dircmd.c:233
#12 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#13 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#14 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#15 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x4659f940 (LWP 2807)):
#0 0x00000035ab00d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00000035ab008c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x00000035ab008afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000031b0a3fc7b in lmgr_p (m=<value optimized out>) at lockmgr.c:95
#4 0x00000031b0a3fded in bthread_mutex_lock_p (m=<value optimized out>,
file=<value optimized out>, line=<value optimized out>) at lockmgr.c:718
#5 0x00000000004277f9 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:222
#6 0x000000000042796b in DEVICE::dblock (this=0x12a68dc0, why=7) at lock.c:145
#7 0x00000000004336c4 in despool_data (dcr=0x12abc3c8, commit=true) at
dev.h:557
#8 0x0000000000434062 in commit_data_spool (dcr=0x12abc3c8) at spool.c:136
#9 0x000000000040f829 in do_append_data (jcr=0x12a96048) at append.c:318
#10 0x0000000000422def in append_data_cmd (jcr=0x12a96048) at fd_cmds.c:203
#11 0x00000000004225a3 in do_fd_commands (jcr=0x12a96048) at fd_cmds.c:162
#12 0x0000000000422d09 in run_job (jcr=0x12a96048) at fd_cmds.c:124
#13 0x0000000000423383 in run_cmd (jcr=0x12a96048) at job.c:225
#14 0x000000000041fbe8 in handle_connection_request (arg=0x12ac1fa8) at
dircmd.c:233
#15 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#16 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#17 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#18 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x46fa0940 (LWP 2808)):
#0 0x00000035ab00ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a3f4f6 in bthread_cond_wait_p (cond=<value optimized out>,
m=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
at lockmgr.c:801
#2 0x0000000000427860 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:233
#3 0x000000000042796b in DEVICE::dblock (this=0x12a68e54, why=7) at lock.c:145
#4 0x00000000004336c4 in despool_data (dcr=0x12abc908, commit=true) at
dev.h:557
#5 0x0000000000434062 in commit_data_spool (dcr=0x12abc908) at spool.c:136
#6 0x000000000040f829 in do_append_data (jcr=0x12ab8548) at append.c:318
#7 0x0000000000422def in append_data_cmd (jcr=0x12ab8548) at fd_cmds.c:203
#8 0x00000000004225a3 in do_fd_commands (jcr=0x12ab8548) at fd_cmds.c:162
#9 0x0000000000422d09 in run_job (jcr=0x12ab8548) at fd_cmds.c:124
#10 0x0000000000423383 in run_cmd (jcr=0x12ab8548) at job.c:225
#11 0x000000000041fbe8 in handle_connection_request (arg=0x12a96568) at
dircmd.c:233
#12 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#13 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#14 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#15 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x479a1940 (LWP 2809)):
#0 0x00000035ab00ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a3f4f6 in bthread_cond_wait_p (cond=<value optimized out>,
m=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
at lockmgr.c:801
#2 0x0000000000427860 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:233
#3 0x000000000042796b in DEVICE::dblock (this=0x12a68e54, why=7) at lock.c:145
#4 0x00000000004336c4 in despool_data (dcr=0x12abce48, commit=true) at
dev.h:557
#5 0x0000000000434062 in commit_data_spool (dcr=0x12abce48) at spool.c:136
#6 0x000000000040f829 in do_append_data (jcr=0x12aba358) at append.c:318
#7 0x0000000000422def in append_data_cmd (jcr=0x12aba358) at fd_cmds.c:203
#8 0x00000000004225a3 in do_fd_commands (jcr=0x12aba358) at fd_cmds.c:162
#9 0x0000000000422d09 in run_job (jcr=0x12aba358) at fd_cmds.c:124
#10 0x0000000000423383 in run_cmd (jcr=0x12aba358) at job.c:225
#11 0x000000000041fbe8 in handle_connection_request (arg=0x12ab9218) at
dircmd.c:233
#12 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#13 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#14 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#15 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x483a2940 (LWP 2811)):
#0 0x00000035ab00d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00000035ab008c3a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x00000035ab008afc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000031b0a3fc7b in lmgr_p (m=<value optimized out>) at lockmgr.c:95
#4 0x00000031b0a3fded in bthread_mutex_lock_p (m=<value optimized out>,
file=<value optimized out>, line=<value optimized out>) at lockmgr.c:718
#5 0x00000000004277f9 in DEVICE::r_dlock (this=0x12a68d88, locked=<value
optimized out>) at lock.c:222
#6 0x000000000042796b in DEVICE::dblock (this=0x12a68dc0, why=7) at lock.c:145
#7 0x00000000004336c4 in despool_data (dcr=0x12abfb28, commit=true) at
dev.h:557
#8 0x0000000000434062 in commit_data_spool (dcr=0x12abfb28) at spool.c:136
#9 0x000000000040f829 in do_append_data (jcr=0x12abe608) at append.c:318
#10 0x0000000000422def in append_data_cmd (jcr=0x12abe608) at fd_cmds.c:203
#11 0x00000000004225a3 in do_fd_commands (jcr=0x12abe608) at fd_cmds.c:162
#12 0x0000000000422d09 in run_job (jcr=0x12abe608) at fd_cmds.c:124
#13 0x0000000000423383 in run_cmd (jcr=0x12abe608) at job.c:225
#14 0x000000000041fbe8 in handle_connection_request (arg=0x12abd918) at
dircmd.c:233
#15 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#16 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#17 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#18 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x4ddab940 (LWP 2840)):
#0 0x00000035ab00ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a30016 in rwl_writelock (rwl=<value optimized out>) at
rwlock.c:242
#2 0x0000000000438217 in _lock_volumes () at vol_mgr.c:112
#3 0x00000000004383f6 in dup_vol_list (jcr=0x650a5c) at vol_mgr.c:766
#4 0x0000000000431645 in find_suitable_device_for_job (jcr=0x12a99c28,
rc...@0x4ddaac10) at reserve.c:430
#5 0x00000000004323c7 in use_cmd (jcr=0x12a99c28) at reserve.c:317
#6 0x000000000041fbe8 in handle_connection_request (arg=0x12a704f8) at
dircmd.c:233
#7 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#8 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#9 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x4d3aa940 (LWP 2845)):
#0 0x00000035ab00ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000031b0a30016 in rwl_writelock (rwl=<value optimized out>) at
rwlock.c:242
#2 0x0000000000438217 in _lock_volumes () at vol_mgr.c:112
#3 0x00000000004387f7 in list_volumes (sendit=0x434ee0 <sendit>,
arg=0x4d3a9cd0) at vol_mgr.c:226
#4 0x00000000004369c6 in output_status (sp=0x4d3a9cd0) at status.c:103
#5 0x0000000000436a90 in status_cmd (jcr=<value optimized out>) at status.c:680
#6 0x000000000041fbe8 in handle_connection_request (arg=0x12a93cc8) at
dircmd.c:233
#7 0x00000031b0a3b5a2 in workq_server (arg=<value optimized out>) at
workq.c:346
#8 0x00000031b0a406d3 in lmgr_thread_launcher (x=<value optimized out>) at
lockmgr.c:882
#9 0x00000035ab006617 in start_thread () from /lib64/libpthread.so.0
#10 0x00000035aa4d3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2ba33b978560 (LWP 9518)):
#0 0x00000035ab00e4af in waitpid () from /lib64/libpthread.so.0
#1 0x00000031b0a31fd0 in signal_handler (sig=<value optimized out>) at
signal.c:229
#2 <signal handler called>
#3 0x00000035aa4cced2 in select () from /lib64/libc.so.6
#4 0x00000031b0a1603b in bnet_thread_server (addrs=<value optimized out>,
max_clients=<value optimized out>, client_wq=<value optimized out>,
handle_client_request=<value optimized out>) at bnet_server.c:161
#5 0x000000000040796d in main (argc=<value optimized out>, argv=<value
optimized out>) at stored.c:312
#0 0x00000035ab00e4af in waitpid () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00000031b0a31fd0 in signal_handler (sig=<value optimized out>) at
signal.c:229
229 waitpid(pid, NULL, 0); /* wait for child to produce dump
*/
Current language: auto; currently c++
sigdefault = Could not find the frame base for "signal_handler".
buf = Could not find the frame base for "signal_handler".
pid = <value optimized out>
exelen = <value optimized out>
already_dead = 1
argv = {0x0, 0x0, 0x0, 0x0, 0x0}
pid_buf = "9518", '\0' <repeats 15 times>
btpath = "/usr/sbin/btraceback", '\0' <repeats 379 times>
#2 <signal handler called>
No symbol table info available.
#3 0x00000035aa4cced2 in select () from /lib64/libc.so.6
No symbol table info available.
#4 0x00000031b0a1603b in bnet_thread_server (addrs=<value optimized out>,
max_clients=<value optimized out>, client_wq=<value optimized out>,
handle_client_request=<value optimized out>) at bnet_server.c:161
161 if ((stat = select(maxfd + 1, &sockset, NULL, NULL, NULL)) < 0) {
maxfd = <value optimized out>
sockset = Could not find the frame base for "bnet_thread_server(dlist*, int,
workq_tag*, void* (*)(void*))".
newsockfd = <value optimized out>
stat = <value optimized out>
clilen = <value optimized out>
cli_addr = Could not find the frame base for "bnet_thread_server(dlist*, int,
workq_tag*, void* (*)(void*))".
tlog = <value optimized out>
turnon = 1
p = <value optimized out>
fd_ptr = <value optimized out>
buf = Could not find the frame base for "bnet_thread_server(dlist*, int,
workq_tag*, void* (*)(void*))".
sockfds = Could not find the frame base for "bnet_thread_server(dlist*, int,
workq_tag*, void* (*)(void*))".
allbuf = Could not find the frame base for "bnet_thread_server(dlist*, int,
workq_tag*, void* (*)(void*))".
#5 0x000000000040796d in main (argc=<value optimized out>, argv=<value
optimized out>) at stored.c:312
312 &dird_workq, handle_connection_request);
ch = <value optimized out>
no_signals = false
test_config = false
thid = 1105692992
uid = 0x7fffb7d3fe8c "bacula"
gid = 0x7fffb7d3fe96 "disk"
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
Attempt to dump locks
threadid=0x4d3aa940 max=1 current=0
lock=0x650a00 state=Wanted priority=10 rwlock.c:237
threadid=0x4ddab940 max=1 current=1
lock=0x6506c0 state=Granted priority=0 rwlock.c:237
lock=0x650a00 state=Wanted priority=10 rwlock.c:237
threadid=0x483a2940 max=3 current=0
lock=0x12a68dc0 state=Wanted priority=5 lock.c:222
threadid=0x479a1940 max=3 current=-1
threadid=0x46fa0940 max=3 current=-1
threadid=0x4659f940 max=3 current=0
lock=0x12a68dc0 state=Wanted priority=5 lock.c:222
threadid=0x45b9e940 max=3 current=-1
threadid=0x4519d940 max=3 current=1
lock=0x12a68dc0 state=Granted priority=5 dev.h:462
lock=0x650a00 state=Granted priority=10 rwlock.c:237
threadid=0x4339a940 max=3 current=0
lock=0x12a68dc0 state=Wanted priority=5 lock.c:222
threadid=0x41960940 max=3 current=-1
threadid=0x43d9b940 max=3 current=0
lock=0x12a68dc0 state=Wanted priority=5 lock.c:222
threadid=0x42999940 max=2 current=-1
threadid=0x2ba33b978560 max=0 current=-1
Attempt to dump current JCRs
JCR=0x12a56ab8 JobId=11447 name=tunicata-etc.2010-03-18_20.32.02_00 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12aa75e8 JobId=11449 name=vs-01-etc.2010-03-18_20.32.02_02 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12a6fb88 JobId=11451 name=vs-02-etc.2010-03-18_20.32.02_04 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12a92d58 JobId=11454 name=eccles-etc.2010-03-18_20.32.02_07 JobStatus=R
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12a946f8 JobId=11456 name=amniota-etc.2010-03-18_20.32.02_09 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12a96048 JobId=11459 name=bcpn-01-etc.2010-03-18_20.32.02_12 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12ab8548 JobId=11462 name=bcpn-02-etc.2010-03-18_20.32.02_15 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12aba358 JobId=11464 name=bcpn-03-etc.2010-03-18_20.32.02_17 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12abe608 JobId=11466 name=noc-etc.2010-03-18_20.32.02_19 JobStatus=l
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=18-Mar-2010 20:32
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12a99c28 JobId=11444 name=agnatha-var.2010-03-18_20.32.01_57 JobStatus=C
use_count=1
JobType=B JobLevel=I
sched_time=18-Mar-2010 20:32 start_time=31-Dec-1969 16:00
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
JCR=0x12a9e548 JobId=0 name=*System* JobStatus=C
use_count=1
JobType=I JobLevel=
sched_time=18-Mar-2010 20:32 start_time=31-Dec-1969 16:00
end_time=31-Dec-1969 16:00 wait_time=31-Dec-1969 16:00
db=(nil) db_batch=(nil) batch_started=0
Attempt to dump plugins. Hook count=0
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
