Dear Ceph-devel, ceph-users,
I am currently facing issue with my ceph mds server. Ceph-mds daemon does not
want to bring up back.
Tried running that manually with ceph-mds -i mon01 -d but it shows that it
stucks at failed assert(session) line 1303 in mds/journal.cc and aborted.
Can someone shed some light in this issue.
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
Let me know if I need to send log with debug enabled.
Regards,
Bazli
________________________________
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s) only and may be
confidential, especially as regards personal data. If you are not the intended
recipient, please note that any dealing, review, distribution, printing,
copying or use of this e-mail is strictly prohibited. If you have received this
email in error, please notify the sender immediately and delete the original
message (including any attachments).
MIMOS Berhad is a research and development institution under the purview of the
Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions
and other information in this e-mail that do not relate to the official
business of MIMOS Berhad and/or its subsidiaries shall be understood as neither
given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
Berhad nor its subsidiaries accepts responsibility for the same. All liability
arising from or in connection with computer viruses and/or corrupted e-mails is
excluded to the fullest extent permitted by law.
------------------------------------------------------------------
-
-
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s)
only and may contain confidential information. If you are not the
intended recipient, please note that any dealing, review,
distribution, printing, copying or use of this e-mail is strictly
prohibited. If you have received this email in error, please notify
the sender immediately and delete the original message.
MIMOS Berhad is a research and development institution under
the purview of the Malaysian Ministry of Science, Technology and
Innovation. Opinions, conclusions and other information in this e-
mail that do not relate to the official business of MIMOS Berhad
and/or its subsidiaries shall be understood as neither given nor
endorsed by MIMOS Berhad and/or its subsidiaries and neither
MIMOS Berhad nor its subsidiaries accepts responsibility for the
same. All liability arising from or in connection with computer
viruses and/or corrupted e-mails is excluded to the fullest extent
permitted by law.
2014-04-25 12:17:27.210367 7f3c30250780 0 ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mds, pid 5492
starting mds.mon01 at :/0
2014-04-25 12:17:27.441530 7f3c2b2d6700 1 mds.-1.0 handle_mds_map standby
2014-04-25 12:17:27.624820 7f3c2b2d6700 1 mds.0.12834 handle_mds_map i am now
mds.0.12834
2014-04-25 12:17:27.624825 7f3c2b2d6700 1 mds.0.12834 handle_mds_map state
change up:standby --> up:replay
2014-04-25 12:17:27.624830 7f3c2b2d6700 1 mds.0.12834 replay_start
2014-04-25 12:17:27.624836 7f3c2b2d6700 1 mds.0.12834 recovery set is
2014-04-25 12:17:27.624837 7f3c2b2d6700 1 mds.0.12834 need osdmap epoch
29082, have 29081
2014-04-25 12:17:27.624839 7f3c2b2d6700 1 mds.0.12834 waiting for osdmap
29082 (which blacklists prior instance)
2014-04-25 12:17:30.138623 7f3c2b2d6700 0 mds.0.cache creating system inode
with ino:100
2014-04-25 12:17:30.138890 7f3c2b2d6700 0 mds.0.cache creating system inode
with ino:1
mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*,
MDSlaveUpdate*)' thread 7f3c26fbd700 time 2014-04-25 12:17:30.441635
mds/journal.cc: 1303: FAILED assert(session)
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
2: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
3: (MDLog::_replay_thread()+0x678) [0x79dbb8]
4: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
5: (()+0x7e9a) [0x7f3c2f675e9a]
6: (clone()+0x6d) [0x7f3c2e56a3fd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
2014-04-25 12:17:30.442489 7f3c26fbd700 -1 mds/journal.cc: In function 'void
EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f3c26fbd700 time
2014-04-25 12:17:30.441635
mds/journal.cc: 1303: FAILED assert(session)
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
2: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
3: (MDLog::_replay_thread()+0x678) [0x79dbb8]
4: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
5: (()+0x7e9a) [0x7f3c2f675e9a]
6: (clone()+0x6d) [0x7f3c2e56a3fd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- begin dump of recent events ---
-172> 2014-04-25 12:17:27.208884 7f3c30250780 5 asok(0x1b99000)
register_command perfcounters_dump hook 0x1b8e010
-171> 2014-04-25 12:17:27.208931 7f3c30250780 5 asok(0x1b99000)
register_command 1 hook 0x1b8e010
-170> 2014-04-25 12:17:27.208935 7f3c30250780 5 asok(0x1b99000)
register_command perf dump hook 0x1b8e010
-169> 2014-04-25 12:17:27.208940 7f3c30250780 5 asok(0x1b99000)
register_command perfcounters_schema hook 0x1b8e010
-168> 2014-04-25 12:17:27.208943 7f3c30250780 5 asok(0x1b99000)
register_command 2 hook 0x1b8e010
-167> 2014-04-25 12:17:27.208945 7f3c30250780 5 asok(0x1b99000)
register_command perf schema hook 0x1b8e010
-166> 2014-04-25 12:17:27.208948 7f3c30250780 5 asok(0x1b99000)
register_command config show hook 0x1b8e010
-165> 2014-04-25 12:17:27.208954 7f3c30250780 5 asok(0x1b99000)
register_command config set hook 0x1b8e010
-164> 2014-04-25 12:17:27.208957 7f3c30250780 5 asok(0x1b99000)
register_command config get hook 0x1b8e010
-163> 2014-04-25 12:17:27.208962 7f3c30250780 5 asok(0x1b99000)
register_command log flush hook 0x1b8e010
-162> 2014-04-25 12:17:27.208965 7f3c30250780 5 asok(0x1b99000)
register_command log dump hook 0x1b8e010
-161> 2014-04-25 12:17:27.208967 7f3c30250780 5 asok(0x1b99000)
register_command log reopen hook 0x1b8e010
-160> 2014-04-25 12:17:27.210367 7f3c30250780 0 ceph version 0.72.2
(a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mds, pid 5492
-159> 2014-04-25 12:17:27.210902 7f3c30250780 1 -- 10.4.118.21:0/0 learned
my addr 10.4.118.21:0/0
-158> 2014-04-25 12:17:27.210915 7f3c30250780 1 accepter.accepter.bind
my_inst.addr is 10.4.118.21:6800/5492 need_addr=0
-157> 2014-04-25 12:17:27.212217 7f3c30250780 5 asok(0x1b99000) init
/var/run/ceph/ceph-mds.mon01.asok
-156> 2014-04-25 12:17:27.212232 7f3c30250780 5 asok(0x1b99000)
bind_and_listen /var/run/ceph/ceph-mds.mon01.asok
-155> 2014-04-25 12:17:27.212267 7f3c30250780 5 asok(0x1b99000)
register_command 0 hook 0x1b8d0b8
-154> 2014-04-25 12:17:27.212278 7f3c30250780 5 asok(0x1b99000)
register_command version hook 0x1b8d0b8
-153> 2014-04-25 12:17:27.212288 7f3c30250780 5 asok(0x1b99000)
register_command git_version hook 0x1b8d0b8
-152> 2014-04-25 12:17:27.212291 7f3c30250780 5 asok(0x1b99000)
register_command help hook 0x1b8e0d0
-151> 2014-04-25 12:17:27.212293 7f3c30250780 5 asok(0x1b99000)
register_command get_command_descriptions hook 0x1b8e0c0
-150> 2014-04-25 12:17:27.212337 7f3c30250780 10 monclient(hunting):
build_initial_monmap
-149> 2014-04-25 12:17:27.212370 7f3c30250780 1 -- 10.4.118.21:6800/5492
messenger.start
-148> 2014-04-25 12:17:27.212421 7f3c30250780 5 adding auth protocol: cephx
-147> 2014-04-25 12:17:27.212429 7f3c30250780 5 adding auth protocol: cephx
-146> 2014-04-25 12:17:27.212419 7f3c2c2d8700 5 asok(0x1b99000) entry start
-145> 2014-04-25 12:17:27.212513 7f3c30250780 1 accepter.accepter.start
-144> 2014-04-25 12:17:27.212539 7f3c30250780 10 monclient(hunting): init
-143> 2014-04-25 12:17:27.212543 7f3c30250780 5 adding auth protocol: cephx
-142> 2014-04-25 12:17:27.212544 7f3c30250780 10 monclient(hunting):
auth_supported 2 method cephx
-141> 2014-04-25 12:17:27.212709 7f3c30250780 2 auth: KeyRing::load: loaded
key file /var/lib/ceph/mds/ceph-mon01/keyring
-140> 2014-04-25 12:17:27.212779 7f3c30250780 10 monclient(hunting):
_reopen_session rank -1 name
-139> 2014-04-25 12:17:27.212822 7f3c30250780 10 monclient(hunting): picked
mon.noname-b con 0x1bb3580 addr 10.4.118.22:6789/0
-138> 2014-04-25 12:17:27.212844 7f3c30250780 10 monclient(hunting):
_send_mon_message to mon.noname-b at 10.4.118.22:6789/0
-137> 2014-04-25 12:17:27.212849 7f3c30250780 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x1b98000 con
0x1bb3580
-136> 2014-04-25 12:17:27.212860 7f3c30250780 10 monclient(hunting):
renew_subs
-135> 2014-04-25 12:17:27.213683 7f3c2b2d6700 5 mds.-1.0 ms_handle_connect
on 10.4.118.22:6789/0
-134> 2014-04-25 12:17:27.214213 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 1 ==== mon_map v1 ==== 485+0+0 (3841125561 0 0)
0x1bd2400 con 0x1bb3580
-133> 2014-04-25 12:17:27.214260 7f3c2b2d6700 10 monclient(hunting):
handle_monmap mon_map v1
-132> 2014-04-25 12:17:27.214277 7f3c2b2d6700 10 monclient(hunting): got
monmap 1, mon.noname-b is now rank -1
-131> 2014-04-25 12:17:27.214279 7f3c2b2d6700 10 monclient(hunting): dump:
epoch 1
fsid 0a2f6c33-b08e-47f3-aece-56037a00250d
last_changed 0.000000
created 0.000000
0: 10.4.118.21:6789/0 mon.mon01
1: 10.4.118.22:6789/0 mon.mon02
2: 10.4.118.23:6789/0 mon.mon03
-130> 2014-04-25 12:17:27.214317 7f3c2b2d6700 1 monclient(hunting): found
mon.mon02
-129> 2014-04-25 12:17:27.214328 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0
(3487474439 0 0) 0x1bd2200 con 0x1bb3580
-128> 2014-04-25 12:17:27.214356 7f3c2b2d6700 10 monclient: my global_id is
1080301
-127> 2014-04-25 12:17:27.214545 7f3c2b2d6700 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-126> 2014-04-25 12:17:27.214556 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1b98900 con
0x1bb3580
-125> 2014-04-25 12:17:27.215290 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0
(172903091 0 0) 0x1bd2600 con 0x1bb3580
-124> 2014-04-25 12:17:27.215402 7f3c2b2d6700 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-123> 2014-04-25 12:17:27.215411 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1b98d80 con
0x1bb3580
-122> 2014-04-25 12:17:27.216180 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0
(1142054316 0 0) 0x1bd2400 con 0x1bb3580
-121> 2014-04-25 12:17:27.216279 7f3c2b2d6700 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-120> 2014-04-25 12:17:27.216287 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1ba5700 con
0x1bb3580
-119> 2014-04-25 12:17:27.216317 7f3c2b2d6700 10 monclient:
_check_auth_rotating renewing rotating keys (they expired before 2014-04-25
12:16:57.216316)
-118> 2014-04-25 12:17:27.216342 7f3c2b2d6700 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-117> 2014-04-25 12:17:27.216346 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1b98b40 con
0x1bb3580
-116> 2014-04-25 12:17:27.216403 7f3c30250780 5 monclient: authenticate
success, global_id 1080301
-115> 2014-04-25 12:17:27.216419 7f3c30250780 10 monclient:
wait_auth_rotating waiting (until 2014-04-25 12:17:57.216417)
-114> 2014-04-25 12:17:27.216658 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 5 ==== mon_map v1 ==== 485+0+0 (3841125561 0 0)
0x1bd2800 con 0x1bb3580
-113> 2014-04-25 12:17:27.216679 7f3c2b2d6700 10 monclient: handle_monmap
mon_map v1
-112> 2014-04-25 12:17:27.216688 7f3c2b2d6700 10 monclient: got monmap 1,
mon.mon02 is now rank 1
-111> 2014-04-25 12:17:27.216690 7f3c2b2d6700 10 monclient: dump:
epoch 1
fsid 0a2f6c33-b08e-47f3-aece-56037a00250d
last_changed 0.000000
created 0.000000
0: 10.4.118.21:6789/0 mon.mon01
1: 10.4.118.22:6789/0 mon.mon02
2: 10.4.118.23:6789/0 mon.mon03
-110> 2014-04-25 12:17:27.216708 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0
(2112033603 0 0) 0x1ba5a80 con 0x1bb3580
-109> 2014-04-25 12:17:27.216720 7f3c2b2d6700 10 monclient:
handle_subscribe_ack sent 2014-04-25 12:17:27.212861 renew after 2014-04-25
12:19:57.212861
-108> 2014-04-25 12:17:27.216898 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0
(3391687514 0 0) 0x1bd2400 con 0x1bb3580
-107> 2014-04-25 12:17:27.216960 7f3c2b2d6700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after 2014-04-25
12:16:57.216960)
-106> 2014-04-25 12:17:27.217008 7f3c30250780 10 monclient:
wait_auth_rotating done
-105> 2014-04-25 12:17:27.217030 7f3c30250780 5 asok(0x1b99000)
register_command objecter_requests hook 0x1b8e120
-104> 2014-04-25 12:17:27.217090 7f3c30250780 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-103> 2014-04-25 12:17:27.217100 7f3c30250780 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- mdsbeacon(1080301/mon01 up:boot seq 1 v0) v2 -- ?+0
0x1bc1dc0 con 0x1bb3580
-102> 2014-04-25 12:17:27.217121 7f3c30250780 10 monclient: renew_subs
-101> 2014-04-25 12:17:27.217125 7f3c30250780 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-100> 2014-04-25 12:17:27.217129 7f3c30250780 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x1ba5540
con 0x1bb3580
-99> 2014-04-25 12:17:27.217139 7f3c30250780 10 monclient: renew_subs
-98> 2014-04-25 12:17:27.217141 7f3c30250780 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-97> 2014-04-25 12:17:27.217146 7f3c30250780 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0
0x1ba5e00 con 0x1bb3580
-96> 2014-04-25 12:17:27.218082 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 8 ==== osd_map(29081..29081 src has 28357..29081) v3
==== 17655+0+0 (1921581891 0 0) 0x1be5000 con 0x1bb3580
-95> 2014-04-25 12:17:27.218264 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0
(2112033603 0 0) 0x1ba5540 con 0x1bb3580
-94> 2014-04-25 12:17:27.218278 7f3c2b2d6700 10 monclient:
handle_subscribe_ack sent 2014-04-25 12:17:27.217125 renew after 2014-04-25
12:19:57.217125
-93> 2014-04-25 12:17:27.218292 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 10 ==== mdsmap(e 40566) v1 ==== 605+0+0 (1596342611 0
0) 0x1bd2600 con 0x1bb3580
-92> 2014-04-25 12:17:27.218305 7f3c2b2d6700 5 mds.-1.0 handle_mds_map
epoch 40566 from mon.1
-91> 2014-04-25 12:17:27.218348 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 11 ==== osd_map(29081..29081 src has 28357..29081) v3
==== 17655+0+0 (1921581891 0 0) 0x1be5480 con 0x1bb3580
-90> 2014-04-25 12:17:27.218363 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0
(2112033603 0 0) 0x1ba5700 con 0x1bb3580
-89> 2014-04-25 12:17:27.218371 7f3c2b2d6700 10 monclient:
handle_subscribe_ack sent 0.000000, ignoring
-88> 2014-04-25 12:17:27.441417 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 13 ==== mdsmap(e 40567) v1 ==== 597+0+0 (2665857825 0
0) 0x1bd2200 con 0x1bb3580
-87> 2014-04-25 12:17:27.441477 7f3c2b2d6700 5 mds.-1.-1 handle_mds_map
epoch 40567 from mon.1
-86> 2014-04-25 12:17:27.441512 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492
mark_down 10.4.118.21:6800/5365 -- pipe dne
-85> 2014-04-25 12:17:27.441530 7f3c2b2d6700 1 mds.-1.0 handle_mds_map
standby
-84> 2014-04-25 12:17:27.624723 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 14 ==== mdsmap(e 40568) v1 ==== 605+0+0 (323821650 0
0) 0x1bd2600 con 0x1bb3580
-83> 2014-04-25 12:17:27.624781 7f3c2b2d6700 5 mds.-1.0 handle_mds_map
epoch 40568 from mon.1
-82> 2014-04-25 12:17:27.624820 7f3c2b2d6700 1 mds.0.12834 handle_mds_map i
am now mds.0.12834
-81> 2014-04-25 12:17:27.624825 7f3c2b2d6700 1 mds.0.12834 handle_mds_map
state change up:standby --> up:replay
-80> 2014-04-25 12:17:27.624830 7f3c2b2d6700 1 mds.0.12834 replay_start
-79> 2014-04-25 12:17:27.624836 7f3c2b2d6700 1 mds.0.12834 recovery set is
-78> 2014-04-25 12:17:27.624837 7f3c2b2d6700 1 mds.0.12834 need osdmap
epoch 29082, have 29081
-77> 2014-04-25 12:17:27.624839 7f3c2b2d6700 1 mds.0.12834 waiting for
osdmap 29082 (which blacklists prior instance)
-76> 2014-04-25 12:17:27.624850 7f3c2b2d6700 10 monclient: renew_subs
-75> 2014-04-25 12:17:27.624854 7f3c2b2d6700 10 monclient: _send_mon_message
to mon.mon02 at 10.4.118.22:6789/0
-74> 2014-04-25 12:17:27.624859 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.22:6789/0 -- mon_subscribe({mdsmap=40569+,monmap=2+,osdmap=29082}) v2
-- ?+0 0x1ba5700 con 0x1bb3580
-73> 2014-04-25 12:17:27.625449 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 15 ==== osd_map(29082..29082 src has 28357..29082) v3
==== 312+0+0 (332989921 0 0) 0x1be5240 con 0x1bb3580
-72> 2014-04-25 12:17:27.625516 7f3c2b2d6700 2 mds.0.12834 boot_start 1:
opening inotable
-71> 2014-04-25 12:17:27.625663 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6812/3522 -- osd_op(mds.0.12834:1 mds0_inotable [read 0~0]
1.b852b893 e29082) v4 -- ?+0 0x1be5480 con 0x1bb3840
-70> 2014-04-25 12:17:27.625684 7f3c2b2d6700 2 mds.0.12834 boot_start 1:
opening sessionmap
-69> 2014-04-25 12:17:27.625727 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6819/244084 -- osd_op(mds.0.12834:2 mds0_sessionmap [read 0~0]
1.3270c60b e29082) v4 -- ?+0 0x1be5000 con 0x1bb3b00
-68> 2014-04-25 12:17:27.625755 7f3c2b2d6700 2 mds.0.12834 boot_start 1:
opening anchor table
-67> 2014-04-25 12:17:27.625821 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6830/3700 -- osd_op(mds.0.12834:3 mds_anchortable [read 0~0]
1.a977f6a7 e29082) v4 -- ?+0 0x1b98000 con 0x1bb39a0
-66> 2014-04-25 12:17:27.625833 7f3c2b2d6700 2 mds.0.12834 boot_start 1:
opening snap table
-65> 2014-04-25 12:17:27.625852 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6819/244084 -- osd_op(mds.0.12834:4 mds_snaptable [read 0~0]
1.d90270ad e29082) v4 -- ?+0 0x1be5d80 con 0x1bb3b00
-64> 2014-04-25 12:17:27.625872 7f3c2b2d6700 2 mds.0.12834 boot_start 1:
opening mds log
-63> 2014-04-25 12:17:27.625877 7f3c2b2d6700 5 mds.0.log open discovering
log bounds
-62> 2014-04-25 12:17:27.625884 7f3c2b2d6700 1 mds.0.journaler(ro) recover
start
-61> 2014-04-25 12:17:27.625885 7f3c2b2d6700 1 mds.0.journaler(ro) read_head
-60> 2014-04-25 12:17:27.625931 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6831/103645 -- osd_op(mds.0.12834:5 200.00000000 [read 0~0]
1.844f3494 e29082) v4 -- ?+0 0x1be5b40 con 0x1bb3dc0
-59> 2014-04-25 12:17:27.625967 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
mon.1 10.4.118.22:6789/0 16 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0
(2112033603 0 0) 0x1ba5540 con 0x1bb3580
-58> 2014-04-25 12:17:27.625975 7f3c2b2d6700 10 monclient:
handle_subscribe_ack sent 2014-04-25 12:17:27.624852 renew after 2014-04-25
12:19:57.624852
-57> 2014-04-25 12:17:27.626972 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.31:6812/3522
-56> 2014-04-25 12:17:27.627001 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.31:6831/103645
-55> 2014-04-25 12:17:27.627007 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6819/244084
-54> 2014-04-25 12:17:27.627048 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6830/3700
-53> 2014-04-25 12:17:27.636626 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.9 10.4.118.31:6831/103645 1 ==== osd_op_reply(5 200.00000000 [read 0~0]
v0'0 uv355974 ondisk = 0) v6 ==== 171+0+84 (3235526312 0 3904168677) 0x1bdd780
con 0x1bb3dc0
-52> 2014-04-25 12:17:27.636672 7f3c2b2d6700 1 mds.0.journaler(ro)
_finish_read_head loghead(trim 539517517824, expire 539517519175, write
1079630739888). probing for end of log (from 1079630739888)...
-51> 2014-04-25 12:17:27.636679 7f3c2b2d6700 1 mds.0.journaler(ro) probing
for end of the log
-50> 2014-04-25 12:17:27.636782 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6833/3627 -- osd_op(mds.0.12834:6 200.0003ed7c [stat] 1.37d52fb2
rwordered e29082) v4 -- ?+0 0x1be5240 con 0x1bb3c60
-49> 2014-04-25 12:17:27.636844 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6812/3663 -- osd_op(mds.0.12834:7 200.0003ed7d [stat] 1.fe6df316
rwordered e29082) v4 -- ?+0 0x1be5900 con 0x1bfc420
-48> 2014-04-25 12:17:27.636866 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.3 10.4.118.31:6812/3522 1 ==== osd_op_reply(1 mds0_inotable [read 0~0] v0'0
uv89057 ondisk = 0) v6 ==== 172+0+34 (1963316113 0 1815378018) 0x1bdda00 con
0x1bb3840
-47> 2014-04-25 12:17:27.637812 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.31:6833/3627
-46> 2014-04-25 12:17:27.637975 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6812/3663
-45> 2014-04-25 12:17:27.644496 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.1 10.4.118.31:6833/3627 1 ==== osd_op_reply(6 200.0003ed7c [stat] v0'0
uv174916 ondisk = 0) v6 ==== 171+0+16 (1154285440 0 661524091) 0x1bfbc80 con
0x1bb3c60
-44> 2014-04-25 12:17:27.673437 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.18 10.4.118.32:6819/244084 1 ==== osd_op_reply(2 mds0_sessionmap [read 0~0]
v0'0 uv38862 ondisk = 0) v6 ==== 174+0+194 (608902333 0 240658729) 0x1bdd780
con 0x1bb3b00
-43> 2014-04-25 12:17:27.673948 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.18 10.4.118.32:6819/244084 2 ==== osd_op_reply(4 mds_snaptable [read 0~0]
v0'0 uv1 ondisk = 0) v6 ==== 172+0+46 (145369538 0 2603725198) 0x1bfb000 con
0x1bb3b00
-42> 2014-04-25 12:17:27.694594 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.16 10.4.118.32:6812/3663 1 ==== osd_op_reply(7 200.0003ed7d [stat] v0'0
uv87037 ondisk = 0) v6 ==== 171+0+16 (3264976615 0 4032609480) 0x1bfb280 con
0x1bfc420
-41> 2014-04-25 12:17:27.694673 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6830/3700 -- osd_op(mds.0.12834:8 200.0003ed7e [stat] 1.8b30ab51
rwordered e29082) v4 -- ?+0 0x1c0bb40 con 0x1bb39a0
-40> 2014-04-25 12:17:27.696126 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.23 10.4.118.32:6830/3700 1 ==== osd_op_reply(8 200.0003ed7e [stat] v0'0
uv172611 ondisk = 0) v6 ==== 171+0+16 (342541003 0 2350581959) 0x1bfb500 con
0x1bb39a0
-39> 2014-04-25 12:17:27.696161 7f3c2b2d6700 1 mds.0.journaler(ro)
_finish_probe_end write_pos = 1079642766914 (header had 1079630739888).
recovered.
-38> 2014-04-25 12:17:29.857399 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.23 10.4.118.32:6830/3700 2 ==== osd_op_reply(3 mds_anchortable [read 0~0]
v0'0 uv85724 ondisk = 0) v6 ==== 174+0+32312596 (727400934 0 462577254)
0x1bfbc80 con 0x1bb39a0
-37> 2014-04-25 12:17:30.138588 7f3c2b2d6700 2 mds.0.12834 boot_start 2:
loading/discovering base inodes
-36> 2014-04-25 12:17:30.138623 7f3c2b2d6700 0 mds.0.cache creating system
inode with ino:100
-35> 2014-04-25 12:17:30.138743 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6812/3522 -- osd_op(mds.0.12834:9 100.00000000 [getxattr inode]
1.c5265ab3 e29082) v4 -- ?+0 0x1c0b6c0 con 0x1bb3840
-34> 2014-04-25 12:17:30.138869 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6801/3498 -- osd_op(mds.0.12834:10 100.00000000.inode [read 0~0]
1.85dde07f e29082) v4 -- ?+0 0x1c0b480 con 0x1bfc2c0
-33> 2014-04-25 12:17:30.138890 7f3c2b2d6700 0 mds.0.cache creating system
inode with ino:1
-32> 2014-04-25 12:17:30.138955 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6815/3667 -- osd_op(mds.0.12834:11 1.00000000 [getxattr inode]
1.6b2cdaff e29082) v4 -- ?+0 0x1c0bd80 con 0x1bfc160
-31> 2014-04-25 12:17:30.139019 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6827/3610 -- osd_op(mds.0.12834:12 1.00000000.inode [read 0~0]
1.232c0e14 e29082) v4 -- ?+0 0x1c0bb40 con 0x1bfc000
-30> 2014-04-25 12:17:30.140168 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.31:6801/3498
-29> 2014-04-25 12:17:30.140361 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.31:6827/3610
-28> 2014-04-25 12:17:30.140376 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6815/3667
-27> 2014-04-25 12:17:30.140689 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.3 10.4.118.31:6812/3522 2 ==== osd_op_reply(9 100.00000000 [getxattr] v0'0
uv0 ondisk = -61 (No data available)) v6 ==== 171+0+0 (2307099340 0 0)
0x71a6280 con 0x1bb3840
-26> 2014-04-25 12:17:30.141488 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.11 10.4.118.31:6801/3498 1 ==== osd_op_reply(10 100.00000000.inode [read
0~0] v0'0 uv85579 ondisk = 0) v6 ==== 177+0+440 (1541854664 0 3546309595)
0x71a6500 con 0x1bfc2c0
-25> 2014-04-25 12:17:30.141713 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.6 10.4.118.31:6827/3610 1 ==== osd_op_reply(12 1.00000000.inode [read 0~0]
v0'0 uv39018 ondisk = 0) v6 ==== 175+0+440 (367118937 0 394604071) 0x71a6780
con 0x1bfc000
-24> 2014-04-25 12:17:30.142011 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.14 10.4.118.32:6815/3667 1 ==== osd_op_reply(11 1.00000000 [getxattr] v0'0
uv0 ondisk = -61 (No data available)) v6 ==== 169+0+0 (826743614 0 0) 0x71a6a00
con 0x1bfc160
-23> 2014-04-25 12:17:30.142062 7f3c2b2d6700 2 mds.0.12834 boot_start 3:
replaying mds log
-22> 2014-04-25 12:17:30.142205 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6815/3667 -- osd_op(mds.0.12834:13 200.0001f677 [read 1351~4192953]
1.558fe013 e29082) v4 -- ?+0 0x71b06c0 con 0x1bfc160
-21> 2014-04-25 12:17:30.142267 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6827/3610 -- osd_op(mds.0.12834:14 200.0001f678 [read 0~4194304]
1.f3f6a32f e29082) v4 -- ?+0 0x71b0b40 con 0x1bfc000
-20> 2014-04-25 12:17:30.142357 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6800/97243 -- osd_op(mds.0.12834:15 200.0001f679 [read 0~4194304]
1.f89d5212 e29082) v4 -- ?+0 0x71b0900 con 0x1bfc580
-19> 2014-04-25 12:17:30.142387 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6819/244084 -- osd_op(mds.0.12834:16 200.0001f67a [read 0~4194304]
1.a5c06abe e29082) v4 -- ?+0 0x1c0b6c0 con 0x1bb3b00
-18> 2014-04-25 12:17:30.142413 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6831/103645 -- osd_op(mds.0.12834:17 200.0001f67b [read 0~4194304]
1.5295be1 e29082) v4 -- ?+0 0x1c0bb40 con 0x1bb3dc0
-17> 2014-04-25 12:17:30.142477 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6806/3602 -- osd_op(mds.0.12834:18 200.0001f67c [read 0~4194304]
1.2ecb82b8 e29082) v4 -- ?+0 0x1be5480 con 0x1bfc840
-16> 2014-04-25 12:17:30.142523 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6806/3602 -- osd_op(mds.0.12834:19 200.0001f67d [read 0~4194304]
1.55323272 e29082) v4 -- ?+0 0x71b0d80 con 0x1bfc840
-15> 2014-04-25 12:17:30.142543 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6830/3700 -- osd_op(mds.0.12834:20 200.0001f67e [read 0~4194304]
1.1155a6e7 e29082) v4 -- ?+0 0x1c0bb40 con 0x1bb39a0
-14> 2014-04-25 12:17:30.142569 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6812/3522 -- osd_op(mds.0.12834:21 200.0001f67f [read 0~4194304]
1.bd437816 e29082) v4 -- ?+0 0x1be5b40 con 0x1bb3840
-13> 2014-04-25 12:17:30.142677 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.32:6827/3696 -- osd_op(mds.0.12834:22 200.0001f680 [read 0~4194304]
1.6b79df46 e29082) v4 -- ?+0 0x1bf2240 con 0x1bfc6e0
-12> 2014-04-25 12:17:30.142709 7f3c26fbd700 1 -- 10.4.118.21:6800/5492 -->
10.4.118.31:6812/3522 -- osd_op(mds.0.12834:23 200.0001f681 [read 0~4194304]
1.a35d67ae e29082) v4 -- ?+0 0x71b6d80 con 0x1bb3840
-11> 2014-04-25 12:17:30.143674 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6806/3602
-10> 2014-04-25 12:17:30.143757 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6800/97243
-9> 2014-04-25 12:17:30.143793 7f3c2b2d6700 5 mds.0.12834
ms_handle_connect on 10.4.118.32:6827/3696
-8> 2014-04-25 12:17:30.212849 7f3c2a2d4700 10 monclient: tick
-7> 2014-04-25 12:17:30.212889 7f3c2a2d4700 10 monclient:
_check_auth_rotating have uptodate secrets (they expire after 2014-04-25
12:17:00.212887)
-6> 2014-04-25 12:17:30.212925 7f3c2a2d4700 10 monclient: renew subs? (now:
2014-04-25 12:17:30.212925; renew after: 2014-04-25 12:19:57.624852) -- no
-5> 2014-04-25 12:17:30.307365 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.9 10.4.118.31:6831/103645 2 ==== osd_op_reply(17 200.0001f67b [read
0~4194304] v0'0 uv78409 ondisk = 0) v6 ==== 171+0+4194304 (2371525742 0
576773970) 0x71b4780 con 0x1bb3dc0
-4> 2014-04-25 12:17:30.339172 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.3 10.4.118.31:6812/3522 3 ==== osd_op_reply(21 200.0001f67f [read
0~4194304] v0'0 uv46142 ondisk = 0) v6 ==== 171+0+4194304 (3069633266 0
4281622723) 0x71a6000 con 0x1bb3840
-3> 2014-04-25 12:17:30.347362 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.6 10.4.118.31:6827/3610 2 ==== osd_op_reply(14 200.0001f678 [read
0~4194304] v0'0 uv89757 ondisk = 0) v6 ==== 171+0+4194304 (2213479130 0
150756498) 0x1bfbc80 con 0x1bfc000
-2> 2014-04-25 12:17:30.440868 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.3 10.4.118.31:6812/3522 4 ==== osd_op_reply(23 200.0001f681 [read
0~4194304] v0'0 uv87746 ondisk = 0) v6 ==== 171+0+4194304 (2711005691 0
924460119) 0x71a6780 con 0x1bb3840
-1> 2014-04-25 12:17:30.441026 7f3c2b2d6700 1 -- 10.4.118.21:6800/5492 <==
osd.14 10.4.118.32:6815/3667 2 ==== osd_op_reply(13 200.0001f677 [read
1351~4192953] v0'0 uv43307 ondisk = 0) v6 ==== 171+0+4192953 (1372967939 0
2121220960) 0x71a6000 con 0x1bfc160
0> 2014-04-25 12:17:30.442489 7f3c26fbd700 -1 mds/journal.cc: In function
'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f3c26fbd700
time 2014-04-25 12:17:30.441635
mds/journal.cc: 1303: FAILED assert(session)
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
2: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
3: (MDLog::_replay_thread()+0x678) [0x79dbb8]
4: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
5: (()+0x7e9a) [0x7f3c2f675e9a]
6: (clone()+0x6d) [0x7f3c2e56a3fd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f3c26fbd700
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: ceph-mds() [0x7e57da]
2: (()+0xfcb0) [0x7f3c2f67dcb0]
3: (gsignal()+0x35) [0x7f3c2e4ac425]
4: (abort()+0x17b) [0x7f3c2e4afb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3c2edff69d]
6: (()+0xb5846) [0x7f3c2edfd846]
7: (()+0xb5873) [0x7f3c2edfd873]
8: (()+0xb596e) [0x7f3c2edfd96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1df) [0x8a2cbf]
10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
11: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
12: (MDLog::_replay_thread()+0x678) [0x79dbb8]
13: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
14: (()+0x7e9a) [0x7f3c2f675e9a]
15: (clone()+0x6d) [0x7f3c2e56a3fd]
2014-04-25 12:17:30.444876 7f3c26fbd700 -1 *** Caught signal (Aborted) **
in thread 7f3c26fbd700
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: ceph-mds() [0x7e57da]
2: (()+0xfcb0) [0x7f3c2f67dcb0]
3: (gsignal()+0x35) [0x7f3c2e4ac425]
4: (abort()+0x17b) [0x7f3c2e4afb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3c2edff69d]
6: (()+0xb5846) [0x7f3c2edfd846]
7: (()+0xb5873) [0x7f3c2edfd873]
8: (()+0xb596e) [0x7f3c2edfd96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1df) [0x8a2cbf]
10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
11: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
12: (MDLog::_replay_thread()+0x678) [0x79dbb8]
13: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
14: (()+0x7e9a) [0x7f3c2f675e9a]
15: (clone()+0x6d) [0x7f3c2e56a3fd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- begin dump of recent events ---
0> 2014-04-25 12:17:30.444876 7f3c26fbd700 -1 *** Caught signal (Aborted)
**
in thread 7f3c26fbd700
ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
1: ceph-mds() [0x7e57da]
2: (()+0xfcb0) [0x7f3c2f67dcb0]
3: (gsignal()+0x35) [0x7f3c2e4ac425]
4: (abort()+0x17b) [0x7f3c2e4afb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f3c2edff69d]
6: (()+0xb5846) [0x7f3c2edfd846]
7: (()+0xb5873) [0x7f3c2edfd873]
8: (()+0xb596e) [0x7f3c2edfd96e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1df) [0x8a2cbf]
10: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
11: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
12: (MDLog::_replay_thread()+0x678) [0x79dbb8]
13: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
14: (()+0x7e9a) [0x7f3c2f675e9a]
15: (clone()+0x6d) [0x7f3c2e56a3fd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
