Re: Can't start ceph mon
On Fri, Nov 23, 2012 at 12:25 AM, Dave (Bob) d...@bob-the-boat.me.uk wrote: Greg, Is there any news on this? David Hurray, there is! Your latest stashed version of the PGMap got corrupted; knowing that helped me to track down a bug — we weren't checking the return value of an fsync! Fixes to prevent this situation in the future are coming shortly; in the mean time I managed to create a replacement PGMap for you that *should* make things right in the world again. Note that it's just blank, so some of your statistics reporting will be wonky for a little while (but all the data is replaceable and will be filled in gradually by the OSDs). Take the attached new_latest file and replace the file at mon.vault01/pgmap/latest with it, then restart your monitor and everything should be good! Let me know if it's not. -Greg new_latest Description: Binary data
Re: Can't start ceph mon
Do you have other monitors in working order? The easiest way to handle it if that's the case is just to remove this monitor from the cluster and add it back in as a new monitor with a fresh store. If not we can look into reconstructing it. -Greg Also, if you still have it, could you zip up your monitor data directory and put it somewhere accessible to us? (I can provide you a drop point if necessary.) We'd like to look at the file layouts a bit since we thought we were properly handling ENOSPC-style issues. -Greg Greg, Thanks. I was running only one monitor, so I will need to try to reconstruct things. If you tell me exactly what to pack up and where to send it, I'll be only too pleased to receive your help... David -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Can't start ceph mon
I have a problem in which I can't start my ceph monitor. The log is shown below.
The log shows version 0.54. I was running 0.52 when the problem arose, and I
moved to the latest in case the newer version fixed the problem.
The original failure happened a week or so ago, and could have been as a result
of running out of disk space when the ceph monitor log became huge.
What should I do to recover the situation?
David
2012-11-19 20:38:51.598468 7fc13fdc6780 0 ceph version 0.54
(commit:60b84b095b1009a305d4d6a5b16f88571cbd3150), process ceph-mon, pid 21012
2012-11-19 20:38:51.598482 7fc13fdc6780 1 store(/ceph/mon.vault01) mount
2012-11-19 20:38:51.598527 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 21
2012-11-19 20:38:51.598542 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
magic = 21 bytes
2012-11-19 20:38:51.598562 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 75
2012-11-19 20:38:51.598567 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.598582 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 205
2012-11-19 20:38:51.598586 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
monmap/latest = 205 bytes
2012-11-19 20:38:51.598809 7fc13fdc6780 1 -- 10.0.1.1:6789/0 learned my addr
10.0.1.1:6789/0
2012-11-19 20:38:51.598818 7fc13fdc6780 1 accepter.accepter.bind my_inst.addr
is 10.0.1.1:6789/0 need_addr=0
2012-11-19 20:38:51.599498 7fc13fdc6780 1 -- 10.0.1.1:6789/0 messenger.start
2012-11-19 20:38:51.599508 7fc13fdc6780 1 accepter.accepter.start
2012-11-19 20:38:51.599610 7fc13fdc6780 1 mon.vault01@-1(probing) e1 init fsid
4d7d8d20-338c-4bdc-9918-9bcf04f9a832
2012-11-19 20:38:51.599674 7fc13cdbe700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c6c0 sd=14 :6789 pgs=0 cs=0 l=0).accept sd=14
2012-11-19 20:38:51.599678 7fc141eff700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c240 sd=9 :6789 pgs=0 cs=0 l=0).accept sd=9
2012-11-19 20:38:51.599718 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 37
2012-11-19 20:38:51.599723 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
cluster_uuid = 37 bytes
2012-11-19 20:38:51.599718 7fc13ccbd700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c480 sd=19 :6789 pgs=0 cs=0 l=0).accept sd=19
2012-11-19 20:38:51.599729 7fc13fdc6780 10 mon.vault01@-1(probing) e1
check_fsid cluster_uuid contains '4d7d8d20-338c-4bdc-9918-9bcf04f9a832'
2012-11-19 20:38:51.599739 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 75
2012-11-19 20:38:51.599745 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.599751 7fc13fdc6780 10 mon.vault01@-1(probing) e1 features
compat={},rocompat={},incompat={1=initial feature set (~v.18)}
2012-11-19 20:38:51.599759 7fc13fdc6780 15 store(/ceph/mon.vault01) exists_bl
joined
2012-11-19 20:38:51.599769 7fc13fdc6780 10 mon.vault01@-1(probing) e1
has_ever_joined = 1
2012-11-19 20:38:51.599794 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/last_committed = 13
2012-11-19 20:38:51.599801 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/first_committed = 132833
2012-11-19 20:38:51.599810 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 239840
2012-11-19 20:38:51.599928 7fc13cbbc700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213cd80 sd=20 :6789 pgs=0 cs=0 l=0).accept sd=20
2012-11-19 20:38:51.599950 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
pgmap/latest = 239840 bytes
--- begin dump of recent events ---2012-11-19 20:38:51.600509 7fc13fdc6780 -1
*** Caught signal (Aborted) **
in thread 7fc13fdc6780
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)
1: ceph-mon() [0x53adf8]
2: (()+0xfe90) [0x7fc141830e90]
3: (gsignal()+0x3e) [0x7fc140016dae]
4: (abort()+0x17b) [0x7fc14001825b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc141af300d]
6: (()+0xb31b6) [0x7fc141af11b6]
7: (()+0xb31e3) [0x7fc141af11e3]
8: (()+0xb32de) [0x7fc141af12de]
9: ceph-mon() [0x5ecb9f]
10: (Paxos::get_stashed(ceph::buffer::list)+0x1ed) [0x49e28d]
11: (Paxos::init()+0x109) [0x49e609]
12: (Monitor::init()+0x36a) [0x485a4a]
13: (main()+0x1289) [0x46d909]
14: (__libc_start_main()+0xed) [0x7fc14000364d]
15: ceph-mon() [0x46fa09]
NOTE: a copy of the executable, or `objdump -rdS executable` is needed to
interpret this.
-55 2012-11-19 20:38:51.596694 7fc13fdc6780 5 asok(0x213d000)
register_command perfcounters_dump hook 0x2131050
-55 2012-11-19 20:38:51.596720 7fc13fdc6780 5 asok(0x213d000)
register_command 1 hook 0x2131050
-54 2012-11-19 20:38:51.596725 7fc13fdc6780 5 asok(0x213d000)
register_command perf dump hook 0x2131050
-53 2012-11-19 20:38:51.596735 7fc13fdc6780 5 asok(0x213d000)
register_command perfcounters_schema hook 0x2131050
-52 2012-11-19 20:38:51.596740 7fc13fdc6780 5 asok(0x213d000)
register_command 2 hook 0x2131050
-51 2012-11-19 20:38:51.596745 7fc13fdc6780 5 asok(0x213d000)
register_command perf schema hook 0x2131050
-50 2012-11-19
Can't Start Ceph Mon
I can't start my ceph monitor, the log is attached below.
Whilst the log shows 0.54, the problem arose with 0.52, and may have been
caused when disk space ran out as a result of a huge set of ceph log files.
Is there a way to recover?
Ragards,
David
bash-4.1# cat /var/log/ceph/mon.vault01.log
2012-11-19 20:38:51.598468 7fc13fdc6780 0 ceph version 0.54
(commit:60b84b095b1009a305d4d6a5b16f88571cbd3150), process ceph-mon, pid 21012
2012-11-19 20:38:51.598482 7fc13fdc6780 1 store(/ceph/mon.vault01) mount
2012-11-19 20:38:51.598527 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 21
2012-11-19 20:38:51.598542 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
magic = 21 bytes
2012-11-19 20:38:51.598562 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 75
2012-11-19 20:38:51.598567 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.598582 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 205
2012-11-19 20:38:51.598586 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
monmap/latest = 205 bytes
2012-11-19 20:38:51.598809 7fc13fdc6780 1 -- 10.0.1.1:6789/0 learned my addr
10.0.1.1:6789/0
2012-11-19 20:38:51.598818 7fc13fdc6780 1 accepter.accepter.bind my_inst.addr
is 10.0.1.1:6789/0 need_addr=0
2012-11-19 20:38:51.599498 7fc13fdc6780 1 -- 10.0.1.1:6789/0 messenger.start
2012-11-19 20:38:51.599508 7fc13fdc6780 1 accepter.accepter.start
2012-11-19 20:38:51.599610 7fc13fdc6780 1 mon.vault01@-1(probing) e1 init fsid
4d7d8d20-338c-4bdc-9918-9bcf04f9a832
2012-11-19 20:38:51.599674 7fc13cdbe700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c6c0 sd=14 :6789 pgs=0 cs=0 l=0).accept sd=14
2012-11-19 20:38:51.599678 7fc141eff700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c240 sd=9 :6789 pgs=0 cs=0 l=0).accept sd=9
2012-11-19 20:38:51.599718 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 37
2012-11-19 20:38:51.599723 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
cluster_uuid = 37 bytes
2012-11-19 20:38:51.599718 7fc13ccbd700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c480 sd=19 :6789 pgs=0 cs=0 l=0).accept sd=19
2012-11-19 20:38:51.599729 7fc13fdc6780 10 mon.vault01@-1(probing) e1
check_fsid cluster_uuid contains '4d7d8d20-338c-4bdc-9918-9bcf04f9a832'
2012-11-19 20:38:51.599739 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 75
2012-11-19 20:38:51.599745 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.599751 7fc13fdc6780 10 mon.vault01@-1(probing) e1 features
compat={},rocompat={},incompat={1=initial feature set (~v.18)}
2012-11-19 20:38:51.599759 7fc13fdc6780 15 store(/ceph/mon.vault01) exists_bl
joined
2012-11-19 20:38:51.599769 7fc13fdc6780 10 mon.vault01@-1(probing) e1
has_ever_joined = 1
2012-11-19 20:38:51.599794 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/last_committed = 13
2012-11-19 20:38:51.599801 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/first_committed = 132833
2012-11-19 20:38:51.599810 7fc13fdc6780 20 store(/ceph/mon.vault01) reading at
off 0 of 239840
2012-11-19 20:38:51.599928 7fc13cbbc700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213cd80 sd=20 :6789 pgs=0 cs=0 l=0).accept sd=20
2012-11-19 20:38:51.599950 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
pgmap/latest = 239840 bytes
--- begin dump of recent events ---2012-11-19 20:38:51.600509 7fc13fdc6780 -1
*** Caught signal (Aborted) **
in thread 7fc13fdc6780
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)
1: ceph-mon() [0x53adf8]
2: (()+0xfe90) [0x7fc141830e90]
3: (gsignal()+0x3e) [0x7fc140016dae]
4: (abort()+0x17b) [0x7fc14001825b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc141af300d]
6: (()+0xb31b6) [0x7fc141af11b6]
7: (()+0xb31e3) [0x7fc141af11e3]
8: (()+0xb32de) [0x7fc141af12de]
9: ceph-mon() [0x5ecb9f]
10: (Paxos::get_stashed(ceph::buffer::list)+0x1ed) [0x49e28d]
11: (Paxos::init()+0x109) [0x49e609]
12: (Monitor::init()+0x36a) [0x485a4a]
13: (main()+0x1289) [0x46d909]
14: (__libc_start_main()+0xed) [0x7fc14000364d]
15: ceph-mon() [0x46fa09]
NOTE: a copy of the executable, or `objdump -rdS executable` is needed to
interpret this.
-55 2012-11-19 20:38:51.596694 7fc13fdc6780 5 asok(0x213d000)
register_command perfcounters_dump hook 0x2131050
-55 2012-11-19 20:38:51.596720 7fc13fdc6780 5 asok(0x213d000)
register_command 1 hook 0x2131050
-54 2012-11-19 20:38:51.596725 7fc13fdc6780 5 asok(0x213d000)
register_command perf dump hook 0x2131050
-53 2012-11-19 20:38:51.596735 7fc13fdc6780 5 asok(0x213d000)
register_command perfcounters_schema hook 0x2131050
-52 2012-11-19 20:38:51.596740 7fc13fdc6780 5 asok(0x213d000)
register_command 2 hook 0x2131050
-51 2012-11-19 20:38:51.596745 7fc13fdc6780 5 asok(0x213d000)
register_command perf schema hook 0x2131050
-50 2012-11-19 20:38:51.596752 7fc13fdc6780 5 asok(0x213d000)
register_command config show hook 0x2131050
-49 2012-11-19 20:38:51.596756
Re: Can't start ceph mon
On Mon, Nov 19, 2012 at 1:08 PM, Dave Humphreys (Datatone)
d...@datatone.co.uk wrote:
I have a problem in which I can't start my ceph monitor. The log is shown
below.
The log shows version 0.54. I was running 0.52 when the problem arose, and I
moved to the latest in case the newer version fixed the problem.
The original failure happened a week or so ago, and could have been as a
result of running out of disk space when the ceph monitor log became huge.
That is almost certainly the case, although I thought we were handling
this issue better now.
What should I do to recover the situation?
Do you have other monitors in working order? The easiest way to handle
it if that's the case is just to remove this monitor from the cluster
and add it back in as a new monitor with a fresh store. If not we can
look into reconstructing it.
-Greg
David
2012-11-19 20:38:51.598468 7fc13fdc6780 0 ceph version 0.54
(commit:60b84b095b1009a305d4d6a5b16f88571cbd3150), process ceph-mon, pid 21012
2012-11-19 20:38:51.598482 7fc13fdc6780 1 store(/ceph/mon.vault01) mount
2012-11-19 20:38:51.598527 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 21
2012-11-19 20:38:51.598542 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
magic = 21 bytes
2012-11-19 20:38:51.598562 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 75
2012-11-19 20:38:51.598567 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.598582 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 205
2012-11-19 20:38:51.598586 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
monmap/latest = 205 bytes
2012-11-19 20:38:51.598809 7fc13fdc6780 1 -- 10.0.1.1:6789/0 learned my addr
10.0.1.1:6789/0
2012-11-19 20:38:51.598818 7fc13fdc6780 1 accepter.accepter.bind
my_inst.addr is 10.0.1.1:6789/0 need_addr=0
2012-11-19 20:38:51.599498 7fc13fdc6780 1 -- 10.0.1.1:6789/0 messenger.start
2012-11-19 20:38:51.599508 7fc13fdc6780 1 accepter.accepter.start
2012-11-19 20:38:51.599610 7fc13fdc6780 1 mon.vault01@-1(probing) e1 init
fsid 4d7d8d20-338c-4bdc-9918-9bcf04f9a832
2012-11-19 20:38:51.599674 7fc13cdbe700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c6c0 sd=14 :6789 pgs=0 cs=0 l=0).accept sd=14
2012-11-19 20:38:51.599678 7fc141eff700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c240 sd=9 :6789 pgs=0 cs=0 l=0).accept sd=9
2012-11-19 20:38:51.599718 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 37
2012-11-19 20:38:51.599723 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
cluster_uuid = 37 bytes
2012-11-19 20:38:51.599718 7fc13ccbd700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c480 sd=19 :6789 pgs=0 cs=0 l=0).accept sd=19
2012-11-19 20:38:51.599729 7fc13fdc6780 10 mon.vault01@-1(probing) e1
check_fsid cluster_uuid contains '4d7d8d20-338c-4bdc-9918-9bcf04f9a832'
2012-11-19 20:38:51.599739 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 75
2012-11-19 20:38:51.599745 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.599751 7fc13fdc6780 10 mon.vault01@-1(probing) e1
features compat={},rocompat={},incompat={1=initial feature set (~v.18)}
2012-11-19 20:38:51.599759 7fc13fdc6780 15 store(/ceph/mon.vault01) exists_bl
joined
2012-11-19 20:38:51.599769 7fc13fdc6780 10 mon.vault01@-1(probing) e1
has_ever_joined = 1
2012-11-19 20:38:51.599794 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/last_committed = 13
2012-11-19 20:38:51.599801 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/first_committed = 132833
2012-11-19 20:38:51.599810 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 239840
2012-11-19 20:38:51.599928 7fc13cbbc700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213cd80 sd=20 :6789 pgs=0 cs=0 l=0).accept sd=20
2012-11-19 20:38:51.599950 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
pgmap/latest = 239840 bytes
--- begin dump of recent events ---2012-11-19 20:38:51.600509 7fc13fdc6780 -1
*** Caught signal (Aborted) **
in thread 7fc13fdc6780
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)
1: ceph-mon() [0x53adf8]
2: (()+0xfe90) [0x7fc141830e90]
3: (gsignal()+0x3e) [0x7fc140016dae]
4: (abort()+0x17b) [0x7fc14001825b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc141af300d]
6: (()+0xb31b6) [0x7fc141af11b6]
7: (()+0xb31e3) [0x7fc141af11e3]
8: (()+0xb32de) [0x7fc141af12de]
9: ceph-mon() [0x5ecb9f]
10: (Paxos::get_stashed(ceph::buffer::list)+0x1ed) [0x49e28d]
11: (Paxos::init()+0x109) [0x49e609]
12: (Monitor::init()+0x36a) [0x485a4a]
13: (main()+0x1289) [0x46d909]
14: (__libc_start_main()+0xed) [0x7fc14000364d]
15: ceph-mon() [0x46fa09]
NOTE: a copy of the executable, or `objdump -rdS executable` is needed to
interpret this.
-55 2012-11-19 20:38:51.596694 7fc13fdc6780 5 asok(0x213d000)
register_command perfcounters_dump hook 0x2131050
-55 2012-11-19 20:38:51.596720
Re: Can't start ceph mon
Also, if you still have it, could you zip up your monitor data
directory and put it somewhere accessible to us? (I can provide you a
drop point if necessary.) We'd like to look at the file layouts a bit
since we thought we were properly handling ENOSPC-style issues.
-Greg
On Mon, Nov 19, 2012 at 1:45 PM, Gregory Farnum g...@inktank.com wrote:
On Mon, Nov 19, 2012 at 1:08 PM, Dave Humphreys (Datatone)
d...@datatone.co.uk wrote:
I have a problem in which I can't start my ceph monitor. The log is shown
below.
The log shows version 0.54. I was running 0.52 when the problem arose, and I
moved to the latest in case the newer version fixed the problem.
The original failure happened a week or so ago, and could have been as a
result of running out of disk space when the ceph monitor log became huge.
That is almost certainly the case, although I thought we were handling
this issue better now.
What should I do to recover the situation?
Do you have other monitors in working order? The easiest way to handle
it if that's the case is just to remove this monitor from the cluster
and add it back in as a new monitor with a fresh store. If not we can
look into reconstructing it.
-Greg
David
2012-11-19 20:38:51.598468 7fc13fdc6780 0 ceph version 0.54
(commit:60b84b095b1009a305d4d6a5b16f88571cbd3150), process ceph-mon, pid
21012
2012-11-19 20:38:51.598482 7fc13fdc6780 1 store(/ceph/mon.vault01) mount
2012-11-19 20:38:51.598527 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 21
2012-11-19 20:38:51.598542 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
magic = 21 bytes
2012-11-19 20:38:51.598562 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 75
2012-11-19 20:38:51.598567 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.598582 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 205
2012-11-19 20:38:51.598586 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
monmap/latest = 205 bytes
2012-11-19 20:38:51.598809 7fc13fdc6780 1 -- 10.0.1.1:6789/0 learned my
addr 10.0.1.1:6789/0
2012-11-19 20:38:51.598818 7fc13fdc6780 1 accepter.accepter.bind
my_inst.addr is 10.0.1.1:6789/0 need_addr=0
2012-11-19 20:38:51.599498 7fc13fdc6780 1 -- 10.0.1.1:6789/0 messenger.start
2012-11-19 20:38:51.599508 7fc13fdc6780 1 accepter.accepter.start
2012-11-19 20:38:51.599610 7fc13fdc6780 1 mon.vault01@-1(probing) e1 init
fsid 4d7d8d20-338c-4bdc-9918-9bcf04f9a832
2012-11-19 20:38:51.599674 7fc13cdbe700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c6c0 sd=14 :6789 pgs=0 cs=0 l=0).accept sd=14
2012-11-19 20:38:51.599678 7fc141eff700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c240 sd=9 :6789 pgs=0 cs=0 l=0).accept sd=9
2012-11-19 20:38:51.599718 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 37
2012-11-19 20:38:51.599723 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
cluster_uuid = 37 bytes
2012-11-19 20:38:51.599718 7fc13ccbd700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213c480 sd=19 :6789 pgs=0 cs=0 l=0).accept sd=19
2012-11-19 20:38:51.599729 7fc13fdc6780 10 mon.vault01@-1(probing) e1
check_fsid cluster_uuid contains '4d7d8d20-338c-4bdc-9918-9bcf04f9a832'
2012-11-19 20:38:51.599739 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 75
2012-11-19 20:38:51.599745 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
feature_set = 75 bytes
2012-11-19 20:38:51.599751 7fc13fdc6780 10 mon.vault01@-1(probing) e1
features compat={},rocompat={},incompat={1=initial feature set (~v.18)}
2012-11-19 20:38:51.599759 7fc13fdc6780 15 store(/ceph/mon.vault01)
exists_bl joined
2012-11-19 20:38:51.599769 7fc13fdc6780 10 mon.vault01@-1(probing) e1
has_ever_joined = 1
2012-11-19 20:38:51.599794 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/last_committed = 13
2012-11-19 20:38:51.599801 7fc13fdc6780 15 store(/ceph/mon.vault01) get_int
pgmap/first_committed = 132833
2012-11-19 20:38:51.599810 7fc13fdc6780 20 store(/ceph/mon.vault01) reading
at off 0 of 239840
2012-11-19 20:38:51.599928 7fc13cbbc700 1 -- 10.0.1.1:6789/0 :/0
pipe(0x213cd80 sd=20 :6789 pgs=0 cs=0 l=0).accept sd=20
2012-11-19 20:38:51.599950 7fc13fdc6780 15 store(/ceph/mon.vault01) get_bl
pgmap/latest = 239840 bytes
--- begin dump of recent events ---2012-11-19 20:38:51.600509 7fc13fdc6780 -1
*** Caught signal (Aborted) **
in thread 7fc13fdc6780
ceph version 0.54 (commit:60b84b095b1009a305d4d6a5b16f88571cbd3150)
1: ceph-mon() [0x53adf8]
2: (()+0xfe90) [0x7fc141830e90]
3: (gsignal()+0x3e) [0x7fc140016dae]
4: (abort()+0x17b) [0x7fc14001825b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fc141af300d]
6: (()+0xb31b6) [0x7fc141af11b6]
7: (()+0xb31e3) [0x7fc141af11e3]
8: (()+0xb32de) [0x7fc141af12de]
9: ceph-mon() [0x5ecb9f]
10: (Paxos::get_stashed(ceph::buffer::list)+0x1ed) [0x49e28d]
11: (Paxos::init()+0x109) [0x49e609]
12: (Monitor::init()+0x36a) [0x485a4a]
13:
