My storage daemon just stopped running. Looking at the messages scrolling
across the around that time I saw this at:
03-Mar 16:01 gyrus-sd: ERROR in lock.c:268 Failed ASSERT: dev->blocked()
/var/log/messages had this in it
Mar 3 16:01:40 gyrus bacula-sd: Bacula interrupted by signal 11:
Segmentation violation
Mar 3 16:01:41 gyrus kernel: bacula-sd[16751] general protection
rip:42e3bb rsp:44f26920 error:0
This is with bacula version 5.0.1, compiled by me running on Centos 5.4
Linux gyrus.case.edu 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 16:12:36 EST 2009
x86_64 x86_64 x86_64 GNU/Linux
I currently only have 1 drive hooked up to my autochanger and only
yesterday I upgraded to 5.0.1
Bacula e-mailed me the traceback shown below. Any ideas what went wrong?
Bob
[Thread debugging using libthread_db enabled]
[New Thread 0x2b5b6afed380 (LWP 9676)]
[New Thread 0x43b25940 (LWP 16899)]
[New Thread 0x46329940 (LWP 16873)]
[New Thread 0x4812c940 (LWP 16866)]
[New Thread 0x48b2d940 (LWP 16819)]
[New Thread 0x44526940 (LWP 16760)]
[New Thread 0x44f27940 (LWP 16751)]
[New Thread 0x4772b940 (LWP 16744)]
[New Thread 0x4100e940 (LWP 16674)]
[New Thread 0x42723940 (LWP 16576)]
[New Thread 0x46d2a940 (LWP 16250)]
[New Thread 0x43124940 (LWP 9680)]
0x00000035802cced2 in select () from /lib64/libc.so.6
$1 = "gyrus-sd", '\0' <repeats 21 times>
$2 = 0xa5b2538 "bacula-sd"
$3 = 0xa5b2578 "/opt/bacula/bin/bacula-sd"
$4 = 0x0
$5 = 0x2b5b6add73fe "5.0.1 (24 February 2010)"
$6 = 0x2b5b6add7417 "x86_64-unknown-linux-gnu"
$7 = 0x2b5b6add7430 "redhat"
$8 = 0x2b5b6add70f5 ""
$9 = "gyrus.case.edu", '\0' <repeats 35 times>
#0 0x00000035802cced2 in select () from /lib64/libc.so.6
#1 0x00002b5b6adaae17 in bnet_thread_server (addrs=0xa5b4398, max_clients=41,
client_wq=0x64ed60,
handle_client_request=0x41ecb0 <handle_connection_request(void*)>)
at bnet_server.c:161
#2 0x00000000004074c9 in main (argc=<value optimized out>,
argv=<value optimized out>) at stored.c:312
Thread 12 (Thread 0x43124940 (LWP 9680)):
#0 0x0000003580e0ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x00002b5b6adcf0cd in watchdog_thread (arg=<value optimized out>)
at watchdog.c:308
#2 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x46d2a940 (LWP 16250)):
#0 0x0000003580e0d174 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003580e08aca in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003580e0898c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002b5b6add33eb in lmgr_p (m=0xa5ccb60) at lockmgr.c:95
#4 0x00000000004268df in DEVICE::r_dlock (this=0xa5ccb28, locked=128)
at lock.c:222
#5 0x000000000041486a in write_block_to_device (dcr=0xa5de6b8) at block.c:351
#6 0x0000000000432690 in despool_data (dcr=0xa5de6b8, commit=true)
at spool.c:298
#7 0x0000000000432ec2 in commit_data_spool (dcr=0xa5de6b8) at spool.c:136
#8 0x000000000040f1b0 in do_append_data (jcr=0xa680eb8) at append.c:318
#9 0x000000000042217f in append_data_cmd (jcr=0xa680eb8) at fd_cmds.c:203
#10 0x0000000000421943 in do_fd_commands (jcr=0xa680eb8) at fd_cmds.c:162
#11 0x00000000004220a9 in run_job (jcr=0xa680eb8) at fd_cmds.c:124
#12 0x00000000004226f4 in run_cmd (jcr=0xa680eb8) at job.c:225
#13 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#14 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#15 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#16 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x42723940 (LWP 16576)):
#0 0x0000003580e0d5cb in read () from /lib64/libpthread.so.0
#1 0x00002b5b6ada9c36 in read_nbytes (bsock=0xa692f28, ptr=0x42722a14 "",
nbytes=4) at bnet.c:80
#2 0x00002b5b6adad98f in BSOCK::recv (this=0xa692f28) at bsock.c:451
#3 0x00002b5b6ada970f in bget_msg (sock=0xa692f28) at bget_msg.c:60
#4 0x000000000040ea60 in do_append_data (jcr=0xa6269b8) at append.c:183
#5 0x000000000042217f in append_data_cmd (jcr=0xa6269b8) at fd_cmds.c:203
#6 0x0000000000421943 in do_fd_commands (jcr=0xa6269b8) at fd_cmds.c:162
#7 0x00000000004220a9 in run_job (jcr=0xa6269b8) at fd_cmds.c:124
#8 0x00000000004226f4 in run_cmd (jcr=0xa6269b8) at job.c:225
#9 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#10 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#11 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#12 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x4100e940 (LWP 16674)):
#0 0x0000003580e0d174 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003580e08aca in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003580e0898c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002b5b6add33eb in lmgr_p (m=0xa5ccb60) at lockmgr.c:95
#4 0x00000000004268df in DEVICE::r_dlock (this=0xa5ccb28, locked=128)
at lock.c:222
#5 0x000000000041486a in write_block_to_device (dcr=0xa67ecf8) at block.c:351
#6 0x0000000000432690 in despool_data (dcr=0xa67ecf8, commit=false)
at spool.c:298
#7 0x000000000043390b in write_block_to_spool_file (dcr=0xa67ecf8)
at spool.c:472
#8 0x000000000041488c in write_block_to_device (dcr=0xa5ccb60) at block.c:345
#9 0x000000000040eafe in do_append_data (jcr=0xa682148) at append.c:199
#10 0x000000000042217f in append_data_cmd (jcr=0xa682148) at fd_cmds.c:203
#11 0x0000000000421943 in do_fd_commands (jcr=0xa682148) at fd_cmds.c:162
#12 0x00000000004220a9 in run_job (jcr=0xa682148) at fd_cmds.c:124
#13 0x00000000004226f4 in run_cmd (jcr=0xa682148) at job.c:225
#14 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#15 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#16 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#17 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x4772b940 (LWP 16744)):
#0 0x0000003580e0d174 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003580e08aca in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003580e0898c in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002b5b6add33eb in lmgr_p (m=0xa5ccb60) at lockmgr.c:95
#4 0x00000000004268df in DEVICE::r_dlock (this=0xa5ccb28, locked=128)
at lock.c:222
#5 0x000000000041486a in write_block_to_device (dcr=0xa608fc8) at block.c:351
#6 0x0000000000432690 in despool_data (dcr=0xa608fc8, commit=false)
at spool.c:298
#7 0x000000000043390b in write_block_to_spool_file (dcr=0xa608fc8)
at spool.c:472
#8 0x000000000041488c in write_block_to_device (dcr=0xa5ccb60) at block.c:345
#9 0x000000000040eafe in do_append_data (jcr=0xa608b28) at append.c:199
#10 0x000000000042217f in append_data_cmd (jcr=0xa608b28) at fd_cmds.c:203
#11 0x0000000000421943 in do_fd_commands (jcr=0xa608b28) at fd_cmds.c:162
#12 0x00000000004220a9 in run_job (jcr=0xa608b28) at fd_cmds.c:124
#13 0x00000000004226f4 in run_cmd (jcr=0xa608b28) at job.c:225
#14 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#15 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#16 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#17 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x44f27940 (LWP 16751)):
#0 0x0000003580e0d5cb in read () from /lib64/libpthread.so.0
#1 0x00002b5b6ada9c36 in read_nbytes (bsock=0xa5b5348,
ptr=0xa7439a8
"AA6]Aô\023\036Aëö-A-Aô6AAAAA6]Aô\023\036A\202ö-A-Aô6AAAAA6]Aô\023\036Axö-A-Aô6AAAAA6]Aô\023\036AWö-A-Aô6AAAAA6]Aô\023\036AÇö-A-Aô6AAAAA6]Aô\023\036A\215ö-A-Aô6AAAAA6]Aô\023\036ACö-A-Aô6AAAAA6]Aô\023\036A¯ö-A-Aô6AAAAA6]Aô\023\036A´ö-A-Aô6AAAAA6]Aô\023\036A\034ö-A-Aô6AAA"...,
nbytes=65536)
at bnet.c:80
#2 0x00002b5b6adadb7d in BSOCK::recv (this=0xa5b5348) at bsock.c:509
#3 0x00002b5b6ada970f in bget_msg (sock=0xa5b5348) at bget_msg.c:60
#4 0x000000000040ea60 in do_append_data (jcr=0xa693058) at append.c:183
#5 0x000000000042217f in append_data_cmd (jcr=0xa693058) at fd_cmds.c:203
#6 0x0000000000421943 in do_fd_commands (jcr=0xa693058) at fd_cmds.c:162
#7 0x00000000004220a9 in run_job (jcr=0xa693058) at fd_cmds.c:124
#8 0x00000000004226f4 in run_cmd (jcr=0xa693058) at job.c:225
#9 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#10 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#11 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#12 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x44526940 (LWP 16760)):
#0 0x0000003580e0e33f in waitpid () from /lib64/libpthread.so.0
#1 0x00002b5b6adc65d6 in signal_handler (sig=11) at signal.c:229
#2 <signal handler called>
#3 0x0000000000426764 in _unblock_device (file=0x43d245 "lock.c", line=155,
dev=0xa5ccb28) at lock.c:268
#4 0x0000000000426ac4 in DEVICE::dunblock (this=0xa5ccb28,
locked=<value optimized out>) at lock.c:155
#5 0x0000000000432a23 in despool_data (dcr=0xa60a478, commit=false)
at spool.c:365
#6 0x000000000043390b in write_block_to_spool_file (dcr=0xa60a478)
at spool.c:472
#7 0x000000000041488c in write_block_to_device (dcr=0x0) at block.c:345
#8 0x000000000040eafe in do_append_data (jcr=0xa5e4728) at append.c:199
#9 0x000000000042217f in append_data_cmd (jcr=0xa5e4728) at fd_cmds.c:203
#10 0x0000000000421943 in do_fd_commands (jcr=0xa5e4728) at fd_cmds.c:162
#11 0x00000000004220a9 in run_job (jcr=0xa5e4728) at fd_cmds.c:124
#12 0x00000000004226f4 in run_cmd (jcr=0xa5e4728) at job.c:225
#13 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#14 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#15 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#16 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x48b2d940 (LWP 16819)):
#0 0x0000003580e0d5cb in read () from /lib64/libpthread.so.0
#1 0x00002b5b6ada9c36 in read_nbytes (bsock=0xa60b878,
ptr=0x48b2ca14 "ÿÿÿÿ¨§c\n", nbytes=4) at bnet.c:80
#2 0x00002b5b6adad98f in BSOCK::recv (this=0xa60b878) at bsock.c:451
#3 0x00002b5b6ada970f in bget_msg (sock=0xa60b878) at bget_msg.c:60
#4 0x000000000040e9e9 in do_append_data (jcr=0xa60ae78) at append.c:151
#5 0x000000000042217f in append_data_cmd (jcr=0xa60ae78) at fd_cmds.c:203
#6 0x0000000000421943 in do_fd_commands (jcr=0xa60ae78) at fd_cmds.c:162
#7 0x00000000004220a9 in run_job (jcr=0xa60ae78) at fd_cmds.c:124
#8 0x00000000004226f4 in run_cmd (jcr=0xa60ae78) at job.c:225
#9 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#10 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#11 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#12 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x4812c940 (LWP 16866)):
#0 0x0000003580e0d5cb in read () from /lib64/libpthread.so.0
#1 0x00002b5b6ada9c36 in read_nbytes (bsock=0xa6e66d8,
ptr=0xa8586bc
"û\204\036ä¦\"
¿nU\234Ús\v\211¼Ú'ÂyÕV\rl1k¬û\037oaV\025_ºPR+&h\r\224i:/lrWU{u\213µßi1óÃZ*^\032LE«Ã
Ñ,îB\031¡K\ah/í\033/×Ó1tÓaZ[µ¼qK$\022Öd±PØ\224½W\022~´\016^ÍÇ\214<I?ÇF¹²×\201\vfï\"0öWÝ.þk½²vÚóW9\234p÷7\236õP\006v\037\034ò\026{¯Á\232ÓL£\226\b\2302\206ã\231\230|\206ddU;6u\...@_\212]gë7âj#[<úë\203\036",
nbytes=65536) at bnet.c:80
#2 0x00002b5b6adadb7d in BSOCK::recv (this=0xa6e66d8) at bsock.c:509
#3 0x00002b5b6ada970f in bget_msg (sock=0xa6e66d8) at bget_msg.c:60
#4 0x000000000040ea60 in do_append_data (jcr=0xa5e1918) at append.c:183
#5 0x000000000042217f in append_data_cmd (jcr=0xa5e1918) at fd_cmds.c:203
#6 0x0000000000421943 in do_fd_commands (jcr=0xa5e1918) at fd_cmds.c:162
#7 0x00000000004220a9 in run_job (jcr=0xa5e1918) at fd_cmds.c:124
#8 0x00000000004226f4 in run_cmd (jcr=0xa5e1918) at job.c:225
#9 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#10 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#11 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#12 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x46329940 (LWP 16873)):
#0 0x0000003580e0d5cb in read () from /lib64/libpthread.so.0
#1 0x00002b5b6ada9c36 in read_nbytes (bsock=0xa5e1718,
ptr=0xa726094
"om/webservices/SharePointPortalServer/PublishedLinksService\"
xmlns:Z=\"urn:schemas-microsoft-com:\" xmlns:st=\"\"
xmlns=\"http://www.w3.org/TR/REC-html40\";>\r\n\r\n<head>\r\n<META
HTTP-EQUIV=\"Content-Type\" CO"..., nbytes=65536) at bnet.c:80
#2 0x00002b5b6adadb7d in BSOCK::recv (this=0xa5e1718) at bsock.c:509
#3 0x00002b5b6ada970f in bget_msg (sock=0xa5e1718) at bget_msg.c:60
#4 0x000000000040ea60 in do_append_data (jcr=0xa67e258) at append.c:183
#5 0x000000000042217f in append_data_cmd (jcr=0xa67e258) at fd_cmds.c:203
#6 0x0000000000421943 in do_fd_commands (jcr=0xa67e258) at fd_cmds.c:162
#7 0x00000000004220a9 in run_job (jcr=0xa67e258) at fd_cmds.c:124
#8 0x00000000004226f4 in run_cmd (jcr=0xa67e258) at job.c:225
#9 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#10 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#11 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#12 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x43b25940 (LWP 16899)):
#0 0x0000003580e0d5cb in read () from /lib64/libpthread.so.0
#1 0x00002b5b6ada9c36 in read_nbytes (bsock=0xa6826e8,
ptr=0x43b24a14 "ÿÿÿÿü", nbytes=4) at bnet.c:80
#2 0x00002b5b6adad98f in BSOCK::recv (this=0xa6826e8) at bsock.c:451
#3 0x00002b5b6ada970f in bget_msg (sock=0xa6826e8) at bget_msg.c:60
#4 0x000000000040e9e9 in do_append_data (jcr=0xa625a38) at append.c:151
#5 0x000000000042217f in append_data_cmd (jcr=0xa625a38) at fd_cmds.c:203
#6 0x0000000000421943 in do_fd_commands (jcr=0xa625a38) at fd_cmds.c:162
#7 0x00000000004220a9 in run_job (jcr=0xa625a38) at fd_cmds.c:124
#8 0x00000000004226f4 in run_cmd (jcr=0xa625a38) at job.c:225
#9 0x000000000041f088 in handle_connection_request (arg=<value optimized out>)
at dircmd.c:233
#10 0x00002b5b6adcf90e in workq_server (arg=<value optimized out>)
at workq.c:346
#11 0x0000003580e064a7 in start_thread () from /lib64/libpthread.so.0
#12 0x00000035802d3c2d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2b5b6afed380 (LWP 9676)):
#0 0x00000035802cced2 in select () from /lib64/libc.so.6
#1 0x00002b5b6adaae17 in bnet_thread_server (addrs=0xa5b4398, max_clients=41,
client_wq=0x64ed60,
handle_client_request=0x41ecb0 <handle_connection_request(void*)>)
at bnet_server.c:161
#2 0x00000000004074c9 in main (argc=<value optimized out>,
argv=<value optimized out>) at stored.c:312
#0 0x00000035802cced2 in select () from /lib64/libc.so.6
No symbol table info available.
#1 0x00002b5b6adaae17 in bnet_thread_server (addrs=0xa5b4398, max_clients=41,
client_wq=0x64ed60,
handle_client_request=0x41ecb0 <handle_connection_request(void*)>)
at bnet_server.c:161
161 if ((stat = select(maxfd + 1, &sockset, NULL, NULL, NULL)) < 0) {
Current language: auto; currently c++
maxfd = <value optimized out>
sockset = {fds_bits = {16, 0 <repeats 15 times>}}
newsockfd = <value optimized out>
stat = <value optimized out>
clilen = 16
cli_addr = {sa_family = 2,
sa_data = "\tc\201\026µç\000\000\000\000\000\000\000"}
tlog = <value optimized out>
turnon = 1
p = (IPADDR *) 0xa5b44a8
fd_ptr = <value optimized out>
buf = "129.22.181.231\000\000 \206\001", '\0' <repeats 13 times>,
"\210¸Ùj[+\000\000èä¸j[+\000\0000vÂÁÿ\177\000\000ÿÿÿÿ", '\0' <repeats 12
times>, "\230Éd\000\000\000\000\000\001", '\0' <repeats 15 times>,
"8M[\n\000\000\000\000\016|ÂÁÿ\177\000\000\000\000\000\000\000\000\000\000\025Íà\1775\000\000"
sockfds = {<SMARTALLOC> = {<No data fields>}, head = 0x7fffc1c26a20,
tail = 0x7fffc1c26a20, loffset = 0, num_items = 1}
allbuf =
"Û\214\222D\000\000\000\000økÂÁÿ\177\000\000\000\000\000\000\000\000\000\000\022\222à\1775",
'\0' <repeats 11 times>, "ø\\\n", '\0' <repeats 24 times>,
"Ä]î{\bN|n[+\000\000 ¶\\\n", '\0' <repeats 20 times>, "à)U\2005\000\000\000
mÂÁ\000\000\000\000ø\\\n\000\000\000\000\200¹\\\n\000\000\000\000èlÂÁÿ\177\000\000
¶\\\n\000\000\000\000\030Â[n[+\000\000H*U\2005\000\000\000ÐkÂÁÿ\177\000\0000\rU\200\000\000\000\000¡\000\000\000\000\000\000\000\220\000\000\000\000\000\000\000\002",
'\0' <repeats 39 times>, "\001\000\000\000\000\000\000\000\210qþj[+"...
#2 0x00000000004074c9 in main (argc=<value optimized out>,
argv=<value optimized out>) at stored.c:312
312 &dird_workq, handle_connection_request);
ch = <value optimized out>
no_signals = false
test_config = false
thid = 1114782016
uid = 0x7fffc1c27c06 "root"
gid = 0x7fffc1c27c0e "bacula"
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#0 0x0000000000000000 in ?? ()
No symbol table info available.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
