Hi,
we have multiple Ceph clusters. One is used as backend for OpenStack
installation for developers - it's here we test Ceph upgrades before we upgrade
prod Ceph clusters. The Ceph cluster is 4 nodes with 12 osds each running
Ubuntu Trusty with latest 3.13 kernel.
This time when upgrading from 0.94.3 to 0.94.5 ceph-mons died during the
upgrade a couple of times. One when we restarted the first monitor, of three,
during the upgrade procedure and the second time when we ran 'ceph osd unset
noout' in the end.
I thought this was a fluke during the upgrade, but ceph-mons seem to segfault
fairly regular now, the day after the upgrade. Corefile doesn't get dumped, so
I have only the log for this strange behaviour. The cluster has been following
the upgrades from firefly to the current Hammer release and has worked flawless
until now.
The cluster produces and work more or less as normal from the users viewpoint.
Except we get segmentation faults in the logfile. Downgrading is a last resort
that I rather not do.
What can cause these errors and how can I fix it is my question.
-Arnulf
The segfault look like this:
Oct 29 14:29:46 95z3zz1 ceph-mon: 0> 2015-10-29 14:29:46.297786
7f908e5af700 -1 *** Caught signal (Segmentation fault) **#012 in thread
7f908e5af700#012#012 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43)#012 1: /usr/bin/ceph-mon()
[0x9adefa]#012 2: (()+0x10340) [0x7f90936b6340]#012 3:
(std::_Rb_tree<std::string, std::pair<std::string const, std::string>,
std::_Select1st<std::pair<std::string const, std::string> >,
std::less<std::string>, std::allocator<std::pair<std::string const,
std::string> > >::find(std::string const&) const+0x25) [0x6518e5]#012 4:
(get_str_map_key(std::map<std::string, std::string, std::less<std::string>,
std::allocator<std::pair<std::string const, std::string> > > const&,
std::string const&, std::string const*)+0x1e) [0x8a002e]#012 5:
(LogMonitor::update_from_paxos(bool*)+0x87a) [0x6b0a5a]#012 6:
(PaxosService::refresh(bool*)+0x19a) [0x60432a]#012 7:
(Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b03db]#012 8:
(Paxos::do_refresh()+0
x2e) [0x5eea5e]#012 9: (Paxos::commit_finish()+0x569) [0x5fbf39]#012 10:
(C_Committed::finish(int)+0x2b) [0x60038b]#012 11: (Context::complete(int)+0x9)
[0x5d4d89]#012 12: (MonitorDBStore::C_DoTransaction::finish(int)+0x8c)
[0x5ff4bc]#012 13: (Context::complete(int)+0x9) [0x5d4d89]#012 14:
(Finisher::finisher_thread_entry()+0x158) [0x717e88]#012 15: (()+0x8182)
[0x7f90936ae182]#012 16: (clone()+0x6d) [0x7f9091c1947d]#012 NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.
Full log of event:
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.697177 7f3154801700 -1
*** Caught signal (Segmentation fault) **#012 in thread 7f3154801700#012#012
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)#012 1:
/usr/bin/ceph-mon() [0x9adefa]#012 2: (()+0x10340) [0x7f3159b63340]#012 3:
(std::_Rb_tree<std::string, std::pair<std::string const, std::string>,
std::_Select1st<std::pair<std::string const, std::string> >,
std::less<std::string>, std::allocator<std::pair<std::string const,
std::string> > >::find(std::string const&) const+0x25) [0x6518e5]#012 4:
(get_str_map_key(std::map<std::string, std::string, std::less<std::string>,
std::allocator<std::pair<std::string const, std::string> > > const&,
std::string const&, std::string const*)+0x1e) [0x8a002e]#012 5:
(LogMonitor::update_from_paxos(bool*)+0x87a) [0x6b0a5a]#012 6:
(PaxosService::refresh(bool*)+0x19a) [0x60432a]#012 7:
(Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b03db]#012 8:
(Paxos::do_refresh()+0x2e) [0x
5eea5e]#012 9: (Paxos::commit_finish()+0x569) [0x5fbf39]#012 10:
(C_Committed::finish(int)+0x2b) [0x60038b]#012 11: (Context::complete(int)+0x9)
[0x5d4d89]#012 12: (MonitorDBStore::C_DoTransaction::finish(int)+0x8c)
[0x5ff4bc]#012 13: (Context::complete(int)+0x9) [0x5d4d89]#012 14:
(Finisher::finisher_thread_entry()+0x158) [0x717e88]#012 15: (()+0x8182)
[0x7f3159b5b182]#012 16: (clone()+0x6d) [0x7f31580c647d]#012 NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 29 14:29:45 95z3zz1 ceph-mon: --- begin dump of recent events ---
Oct 29 14:29:45 95z3zz1 ceph-mon: -450> 2015-10-29 14:29:44.484656
7f315aa5d8c0 5 asok(0x4daa000) register_command perfcounters_dump hook
0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -449> 2015-10-29 14:29:44.484677
7f315aa5d8c0 5 asok(0x4daa000) register_command 1 hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -448> 2015-10-29 14:29:44.484681
7f315aa5d8c0 5 asok(0x4daa000) register_command perf dump hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -447> 2015-10-29 14:29:44.484686
7f315aa5d8c0 5 asok(0x4daa000) register_command perfcounters_schema hook
0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -446> 2015-10-29 14:29:44.484688
7f315aa5d8c0 5 asok(0x4daa000) register_command 2 hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -445> 2015-10-29 14:29:44.484690
7f315aa5d8c0 5 asok(0x4daa000) register_command perf schema hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -444> 2015-10-29 14:29:44.484692
7f315aa5d8c0 5 asok(0x4daa000) register_command perf reset hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -443> 2015-10-29 14:29:44.484694
7f315aa5d8c0 5 asok(0x4daa000) register_command config show hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -442> 2015-10-29 14:29:44.484697
7f315aa5d8c0 5 asok(0x4daa000) register_command config set hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -441> 2015-10-29 14:29:44.484698
7f315aa5d8c0 5 asok(0x4daa000) register_command config get hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -440> 2015-10-29 14:29:44.484700
7f315aa5d8c0 5 asok(0x4daa000) register_command config diff hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -439> 2015-10-29 14:29:44.484702
7f315aa5d8c0 5 asok(0x4daa000) register_command log flush hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -438> 2015-10-29 14:29:44.484704
7f315aa5d8c0 5 asok(0x4daa000) register_command log dump hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -437> 2015-10-29 14:29:44.484706
7f315aa5d8c0 5 asok(0x4daa000) register_command log reopen hook 0x4d32050
Oct 29 14:29:45 95z3zz1 ceph-mon: -436> 2015-10-29 14:29:44.486767
7f315aa5d8c0 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43),
process ceph-mon, pid 143413
Oct 29 14:29:45 95z3zz1 ceph-mon: -435> 2015-10-29 14:29:44.488255
7f315aa5d8c0 5 asok(0x4daa000) init /var/run/ceph/ceph-mon.95z3zz1.asok
Oct 29 14:29:45 95z3zz1 ceph-mon: -434> 2015-10-29 14:29:44.488271
7f315aa5d8c0 5 asok(0x4daa000) bind_and_listen
/var/run/ceph/ceph-mon.95z3zz1.asok
Oct 29 14:29:45 95z3zz1 ceph-mon: -433> 2015-10-29 14:29:44.488320
7f315aa5d8c0 5 asok(0x4daa000) register_command 0 hook 0x4d2e0b8
Oct 29 14:29:45 95z3zz1 ceph-mon: -432> 2015-10-29 14:29:44.488327
7f315aa5d8c0 5 asok(0x4daa000) register_command version hook 0x4d2e0b8
Oct 29 14:29:45 95z3zz1 ceph-mon: -431> 2015-10-29 14:29:44.488332
7f315aa5d8c0 5 asok(0x4daa000) register_command git_version hook 0x4d2e0b8
Oct 29 14:29:45 95z3zz1 ceph-mon: -430> 2015-10-29 14:29:44.488336
7f315aa5d8c0 5 asok(0x4daa000) register_command help hook 0x4d32150
Oct 29 14:29:45 95z3zz1 ceph-mon: -429> 2015-10-29 14:29:44.488340
7f315aa5d8c0 5 asok(0x4daa000) register_command get_command_descriptions hook
0x4d32140
Oct 29 14:29:45 95z3zz1 ceph-mon: -428> 2015-10-29 14:29:44.488570
7f3155ddf700 5 asok(0x4daa000) entry start
Oct 29 14:29:45 95z3zz1 ceph-mon: -427> 2015-10-29 14:29:44.513886
7f315aa5d8c0 0 starting mon.95z3zz1 rank 0 at 157.249.160.11:6789/0 mon_data
/var/lib/ceph/mon/ceph-95z3zz1 fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: -426> 2015-10-29 14:29:44.513983
7f315aa5d8c0 1 -- 157.249.160.11:6789/0 learned my addr 157.249.160.11:6789/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -425> 2015-10-29 14:29:44.513989
7f315aa5d8c0 1 accepter.accepter.bind my_inst.addr is 157.249.160.11:6789/0
need_addr=0
Oct 29 14:29:45 95z3zz1 ceph-mon: -424> 2015-10-29 14:29:44.514048
7f315aa5d8c0 5 adding auth protocol: cephx
Oct 29 14:29:45 95z3zz1 ceph-mon: -423> 2015-10-29 14:29:44.514055
7f315aa5d8c0 5 adding auth protocol: cephx
Oct 29 14:29:45 95z3zz1 ceph-mon: -422> 2015-10-29 14:29:44.514075
7f315aa5d8c0 10 log_channel(cluster) update_config to_monitors: true to_syslog:
true syslog_facility: daemon prio: info)
Oct 29 14:29:45 95z3zz1 ceph-mon: -421> 2015-10-29 14:29:44.514081
7f315aa5d8c0 10 log_channel(audit) update_config to_monitors: true to_syslog:
true syslog_facility: local0 prio: info)
Oct 29 14:29:45 95z3zz1 ceph-mon: -420> 2015-10-29 14:29:44.514144
7f315aa5d8c0 1 mon.95z3zz1@-1(probing) e7 preinit fsid
6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: -419> 2015-10-29 14:29:44.514476
7f315aa5d8c0 1 mon.95z3zz1@-1(probing).paxosservice(pgmap 33661126..33661653)
refresh upgraded, format 0 -> 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -418> 2015-10-29 14:29:44.514491
7f315aa5d8c0 1 mon.95z3zz1@-1(probing).pg v0 on_upgrade discarding in-core
PGMap
Oct 29 14:29:45 95z3zz1 ceph-mon: -417> 2015-10-29 14:29:44.541117
7f3155ddf700 5 asok(0x4daa000) AdminSocket: request 'get_command_descriptions'
'' to 0x4d32140 returned 1496 bytes
Oct 29 14:29:45 95z3zz1 ceph-mon: -416> 2015-10-29 14:29:44.626420
7f315aa5d8c0 4 mon.95z3zz1@-1(probing).mds e89 new map
Oct 29 14:29:45 95z3zz1 ceph-mon: -415> 2015-10-29 14:29:44.626431
7f315aa5d8c0 0 mon.95z3zz1@-1(probing).mds e89
print_map#012epoch#01189#012flags#0110#012created#0112014-07-31
11:09:25.038824#012modified#0112015-10-28
11:59:17.337799#012tableserver#0110#012root#0110#012session_timeout#01160#012session_autoclose#011300#012max_file_size#0111099511627776#012last_failure#0110#012last_failure_osd_epoch#01161286#012compat#011compat={},rocompat={},incompat={1=base
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor
table}#012max_mds#0111#012in#0110#012up#011{0=15517917}#012failed#011#012stopped#011#012data_pools#0110#012metadata_pool#0111#012inline_data#011disabled#01215517917:#011157.249.160.107:6800/3998
'56z3zz1' mds.0.13 up:active seq 188
Oct 29 14:29:45 95z3zz1 ceph-mon: -414> 2015-10-29 14:29:44.626991
7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features
1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -413> 2015-10-29 14:29:44.627009
7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features
1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -412> 2015-10-29 14:29:44.627013
7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features
1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -411> 2015-10-29 14:29:44.627015
7f315aa5d8c0 0 mon.95z3zz1@-1(probing).osd e61385 crush map has features
1107558400, adjusting msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: -410> 2015-10-29 14:29:44.627470
7f315aa5d8c0 1 mon.95z3zz1@-1(probing).paxosservice(auth 28285..28398) refresh
upgraded, format 0 -> 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -409> 2015-10-29 14:29:44.627977
7f315aa5d8c0 2 auth: KeyRing::load: loaded key file
/var/lib/ceph/mon/ceph-95z3zz1/keyring
Oct 29 14:29:45 95z3zz1 ceph-mon: -408> 2015-10-29 14:29:44.627983
7f315aa5d8c0 5 asok(0x4daa000) register_command mon_status hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -407> 2015-10-29 14:29:44.628001
7f315aa5d8c0 5 asok(0x4daa000) register_command quorum_status hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -406> 2015-10-29 14:29:44.628005
7f315aa5d8c0 5 asok(0x4daa000) register_command sync_force hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -405> 2015-10-29 14:29:44.628007
7f315aa5d8c0 5 asok(0x4daa000) register_command add_bootstrap_peer_hint hook
0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -404> 2015-10-29 14:29:44.628010
7f315aa5d8c0 5 asok(0x4daa000) register_command quorum enter hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -403> 2015-10-29 14:29:44.628012
7f315aa5d8c0 5 asok(0x4daa000) register_command quorum exit hook 0x5e1dbe0
Oct 29 14:29:45 95z3zz1 ceph-mon: -402> 2015-10-29 14:29:44.628023
7f315aa5d8c0 1 -- 157.249.160.11:6789/0 messenger.start
Oct 29 14:29:45 95z3zz1 ceph-mon: -401> 2015-10-29 14:29:44.628058
7f315aa5d8c0 2 mon.95z3zz1@-1(probing) e7 init
Oct 29 14:29:45 95z3zz1 ceph-mon: -400> 2015-10-29 14:29:44.628118
7f315aa5d8c0 1 accepter.accepter.start
Oct 29 14:29:45 95z3zz1 ceph-mon: -399> 2015-10-29 14:29:44.628168
7f315aa5d8c0 0 mon.95z3zz1@-1(probing) e7 my rank is now 0 (was -1)
Oct 29 14:29:45 95z3zz1 ceph-mon: -398> 2015-10-29 14:29:44.628175
7f315aa5d8c0 1 -- 157.249.160.11:6789/0 mark_down_all
Oct 29 14:29:45 95z3zz1 ceph-mon: -397> 2015-10-29 14:29:44.628196
7f315aa5d8c0 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
mon_probe(probe 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name 95z3zz1) v6 -- ?+0
0x4dbac80
Oct 29 14:29:45 95z3zz1 ceph-mon: -396> 2015-10-29 14:29:44.628303
7f315aa5d8c0 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
mon_probe(probe 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name 95z3zz1) v6 -- ?+0
0x4dbaa00
Oct 29 14:29:45 95z3zz1 ceph-mon: -395> 2015-10-29 14:29:44.628313
7f314f84b700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ee2000 sd=13 :6789 s=0
pgs=0 cs=0 l=0 c=0x4d54260).accept sd=13 157.249.160.106:43105/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -394> 2015-10-29 14:29:44.628376
7f314f649700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ef1000 sd=14 :6789 s=0
pgs=0 cs=0 l=0 c=0x4d543c0).accept sd=14 157.249.160.107:56748/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -393> 2015-10-29 14:29:44.628435
7f314f447700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5eec000 sd=16 :6789 s=0
pgs=0 cs=0 l=0 c=0x4d547e0).accept sd=16 172.16.6.12:34085/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -392> 2015-10-29 14:29:44.628696
7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0
pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept connect_seq 2
vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -391> 2015-10-29 14:29:44.628711
7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0
pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept we reset
(peer sent cseq 2, 0x5ee7000.cseq = 0), sending RESETSESSION
Oct 29 14:29:45 95z3zz1 ceph-mon: -390> 2015-10-29 14:29:44.628723
7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0
pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept connect_seq 1
vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -389> 2015-10-29 14:29:44.628737
7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0
pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept we reset
(peer sent cseq 1, 0x5efb000.cseq = 0), sending RESETSESSION
Oct 29 14:29:45 95z3zz1 ceph-mon: -388> 2015-10-29 14:29:44.629016
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4878761
172.16.6.12:0/2056508 1 ==== auth(proto 0 35 bytes epoch 7) v1 ==== 65+0+0
(3944302535 0 0) 0x4f8c900 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -387> 2015-10-29 14:29:44.629055
7f315104e700 5 mon.95z3zz1@0(probing) e7 waitlisting message auth(proto 0 35
bytes epoch 7) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -386> 2015-10-29 14:29:44.629056
7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0
pipe(0x5ee2000 sd=13 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d54260).accept connect_seq 0
vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -385> 2015-10-29 14:29:44.629076
7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0
pipe(0x5ef1000 sd=14 :6789 s=0 pgs=0 cs=0 l=0 c=0x4d543c0).accept connect_seq 0
vs existing 0 state connecting
Oct 29 14:29:45 95z3zz1 ceph-mon: -384> 2015-10-29 14:29:44.629187
7f314f649700 2 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0
pipe(0x5ef1000 sd=14 :6789 s=4 pgs=0 cs=0 l=0 c=0x4d543c0).fault (0) Success
Oct 29 14:29:45 95z3zz1 ceph-mon: -383> 2015-10-29 14:29:44.629247
7f314f84b700 2 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0
pipe(0x5ee2000 sd=13 :6789 s=4 pgs=0 cs=0 l=0 c=0x4d54260).fault (0) Success
Oct 29 14:29:45 95z3zz1 ceph-mon: -382> 2015-10-29 14:29:44.629259
7f314f548700 2 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0
pipe(0x5efb000 sd=17 :42956 s=1 pgs=0 cs=0 l=0 c=0x4d53b80). got
newly_acked_seq 0 vs out_seq 0
Oct 29 14:29:45 95z3zz1 ceph-mon: -381> 2015-10-29 14:29:44.629322
7f314f74a700 2 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0
pipe(0x5ee7000 sd=15 :57775 s=1 pgs=0 cs=0 l=0 c=0x4d53e40). got
newly_acked_seq 0 vs out_seq 0
Oct 29 14:29:45 95z3zz1 ceph-mon: -380> 2015-10-29 14:29:44.629581
7f314f649700 0 -- 157.249.160.11:6789/0 >> 157.249.160.107:6789/0
pipe(0x5efb000 sd=17 :42956 s=2 pgs=3465 cs=1 l=0 c=0x4d53b80).reader missed
message? skipped from seq 0 to 1506138999
Oct 29 14:29:45 95z3zz1 ceph-mon: -379> 2015-10-29 14:29:44.629607
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506138999 ==== mon_probe(reply 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name
56z3zz1 quorum 0,1,2 paxos( fc 62203323 lc 62204013 )) v6 ==== 563+0+0
(1924762453 0 0) 0x4dbaa00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -378> 2015-10-29 14:29:44.629635
7f315104e700 5 mon.95z3zz1@0(probing) e7 setting monitor caps on this
connection
Oct 29 14:29:45 95z3zz1 ceph-mon: -377> 2015-10-29 14:29:44.629654
7f314f84b700 0 -- 157.249.160.11:6789/0 >> 157.249.160.106:6789/0
pipe(0x5ee7000 sd=15 :57775 s=2 pgs=2476 cs=1 l=0 c=0x4d53e40).reader missed
message? skipped from seq 0 to 13999349
Oct 29 14:29:45 95z3zz1 ceph-mon: -376> 2015-10-29 14:29:44.629693
7f315104e700 0 log_channel(cluster) log [INF] : mon.95z3zz1 calling new
monitor election
Oct 29 14:29:45 95z3zz1 ceph-mon: -375> 2015-10-29 14:29:44.629699
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -374> 2015-10-29 14:29:44.629725
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -373> 2015-10-29 14:29:44.629727
7f315104e700 10 log_client log_queue is 1 last_log 1 sent 0 num 1 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -372> 2015-10-29 14:29:44.629730
7f315104e700 10 log_client will send 2015-10-29 14:29:44.629699 mon.0
157.249.160.11:6789/0 1 : cluster [INF] mon.95z3zz1 calling new monitor election
Oct 29 14:29:45 95z3zz1 ceph-mon: -371> 2015-10-29 14:29:44.629745
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -370> 2015-10-29 14:29:44.629769
7f315104e700 5 mon.95z3zz1@0(electing).elector(0) start -- can i be leader?
Oct 29 14:29:45 95z3zz1 ceph-mon: -369> 2015-10-29 14:29:44.629922
7f315104e700 1 mon.95z3zz1@0(electing).elector(424) init, last seen epoch 424
Oct 29 14:29:45 95z3zz1 ceph-mon: -368> 2015-10-29 14:29:44.630487
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 propose 425) v5 -- ?+0 0x4e11e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -367> 2015-10-29 14:29:44.630526
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 propose 425) v5 -- ?+0 0x4e123c0
Oct 29 14:29:45 95z3zz1 ceph-mon: -366> 2015-10-29 14:29:44.630543
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999349 ==== mon_probe(reply 6a57f77c-a465-4fc6-b0a2-93c20371bab8 name c6z3zz1
quorum 0,1,2 paxos( fc 62203323 lc 62204013 )) v6 ==== 563+0+0 (534286656 0 0)
0x4dbac80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -365> 2015-10-29 14:29:44.630555
7f315104e700 5 mon.95z3zz1@0(electing) e7 setting monitor caps on this
connection
Oct 29 14:29:45 95z3zz1 ceph-mon: -364> 2015-10-29 14:29:44.630564
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -363> 2015-10-29 14:29:44.630572
7f315104e700 5 mon.95z3zz1@0(electing) e7 setting monitor caps on this
connection
Oct 29 14:29:45 95z3zz1 ceph-mon: -362> 2015-10-29 14:29:44.630581
7f315104e700 5 mon.95z3zz1@0(electing).paxos(paxos recovering c
62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.630582
lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -361> 2015-10-29 14:29:44.630590
7f315104e700 5 mon.95z3zz1@0(electing).paxos(paxos recovering c
62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.630590
lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -360> 2015-10-29 14:29:44.631261
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999350 ==== election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 ack 425) v5 ====
26181+0+0 (4009097990 0 0) 0x4e11e40 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -359> 2015-10-29 14:29:44.631282
7f315104e700 5 mon.95z3zz1@0(electing).elector(425) handle_ack from mon.1
Oct 29 14:29:45 95z3zz1 ceph-mon: -358> 2015-10-29 14:29:44.631289
7f315104e700 5 mon.95z3zz1@0(electing).elector(425) so far i have
{0=37154696925806591,1=37154696925806591}
Oct 29 14:29:45 95z3zz1 ceph-mon: -357> 2015-10-29 14:29:44.631295
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139000 ==== election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 ack 425) v5 ====
26181+0+0 (4009097990 0 0) 0x4e123c0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -356> 2015-10-29 14:29:44.631307
7f315104e700 5 mon.95z3zz1@0(electing).elector(425) handle_ack from mon.2
Oct 29 14:29:45 95z3zz1 ceph-mon: -355> 2015-10-29 14:29:44.631310
7f315104e700 5 mon.95z3zz1@0(electing).elector(425) so far i have
{0=37154696925806591,1=37154696925806591,2=37154696925806591}
Oct 29 14:29:45 95z3zz1 ceph-mon: -354> 2015-10-29 14:29:44.631514
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 victory 426) v5 -- ?+0 0x4e11e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -353> 2015-10-29 14:29:44.631533
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
election(6a57f77c-a465-4fc6-b0a2-93c20371bab8 victory 426) v5 -- ?+0 0x4e12100
Oct 29 14:29:45 95z3zz1 ceph-mon: -352> 2015-10-29 14:29:44.631549
7f315104e700 0 log_channel(cluster) log [INF] : mon.95z3zz1@0 won leader
election with quorum 0,1,2
Oct 29 14:29:45 95z3zz1 ceph-mon: -351> 2015-10-29 14:29:44.631552
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -350> 2015-10-29 14:29:44.631574
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -349> 2015-10-29 14:29:44.631575
7f315104e700 10 log_client log_queue is 2 last_log 2 sent 1 num 2 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -348> 2015-10-29 14:29:44.631577
7f315104e700 10 log_client will send 2015-10-29 14:29:44.631552 mon.0
157.249.160.11:6789/0 2 : cluster [INF] mon.95z3zz1@0 won leader election with
quorum 0,1,2
Oct 29 14:29:45 95z3zz1 ceph-mon: -347> 2015-10-29 14:29:44.631582
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8c000 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -346> 2015-10-29 14:29:44.631797
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(collect lc 62204013 fc 62203323 pn 21100 opn 0) v3 -- ?+0 0x4dbac80
Oct 29 14:29:45 95z3zz1 ceph-mon: -345> 2015-10-29 14:29:44.631811
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(collect lc 62204013 fc 62203323 pn 21100 opn 0) v3 -- ?+0 0x4dbaa00
Oct 29 14:29:45 95z3zz1 ceph-mon: -344> 2015-10-29 14:29:44.631836
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c
62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.631837
lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -343> 2015-10-29 14:29:44.631843
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c
62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.631844
lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -342> 2015-10-29 14:29:44.631864
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
time_check( ping e 426 r 1 ) v1 -- ?+0 0x4f8cd80
Oct 29 14:29:45 95z3zz1 ceph-mon: -341> 2015-10-29 14:29:44.631875
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
time_check( ping e 426 r 1 ) v1 -- ?+0 0x4f8db00
Oct 29 14:29:45 95z3zz1 ceph-mon: -340> 2015-10-29 14:29:44.634897
7f315104e700 0 log_channel(cluster) log [INF] : HEALTH_WARN; noout flag(s) set
Oct 29 14:29:45 95z3zz1 ceph-mon: -339> 2015-10-29 14:29:44.634903
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -338> 2015-10-29 14:29:44.634927
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -337> 2015-10-29 14:29:44.634929
7f315104e700 10 log_client log_queue is 3 last_log 3 sent 2 num 3 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -336> 2015-10-29 14:29:44.634931
7f315104e700 10 log_client will send 2015-10-29 14:29:44.634903 mon.0
157.249.160.11:6789/0 3 : cluster [INF] HEALTH_WARN; noout flag(s) set
Oct 29 14:29:45 95z3zz1 ceph-mon: -335> 2015-10-29 14:29:44.634938
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8e1c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -334> 2015-10-29 14:29:44.634955
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8c000 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -333> 2015-10-29 14:29:44.634962
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c
62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.634962
lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -332> 2015-10-29 14:29:44.634966
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos recovering c
62203323..62204013) is_readable = 0 - now=2015-10-29 14:29:44.634967
lease_expire=0.000000 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -331> 2015-10-29 14:29:44.634970
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999374 ==== paxos(last lc 62204013 fc 62203323 pn 21100 opn 0) v3 ==== 80+0+0
(1023772294 0 0) 0x5f96d00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -330> 2015-10-29 14:29:44.635016
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139018 ==== paxos(last lc 62204013 fc 62203323 pn 21100 opn 0) v3 ====
80+0+0 (3238039421 0 0) 0x5f93480 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -329> 2015-10-29 14:29:44.635043
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(lease lc 62204013 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x5f96d00
Oct 29 14:29:45 95z3zz1 ceph-mon: -328> 2015-10-29 14:29:44.635057
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(lease lc 62204013 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x5f66280
Oct 29 14:29:45 95z3zz1 ceph-mon: -327> 2015-10-29 14:29:44.635372
7f315104e700 0 log_channel(cluster) log [INF] : monmap e7: 3 mons at
{56z3zz1=157.249.160.107:6789/0,95z3zz1=157.249.160.11:6789/0,c6z3zz1=157.249.160.106:6789/0}
Oct 29 14:29:45 95z3zz1 ceph-mon: -326> 2015-10-29 14:29:44.635376
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -325> 2015-10-29 14:29:44.635402
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -324> 2015-10-29 14:29:44.635404
7f315104e700 10 log_client log_queue is 4 last_log 4 sent 3 num 4 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -323> 2015-10-29 14:29:44.635405
7f315104e700 10 log_client will send 2015-10-29 14:29:44.635376 mon.0
157.249.160.11:6789/0 4 : cluster [INF] monmap e7: 3 mons at
{56z3zz1=157.249.160.107:6789/0,95z3zz1=157.249.160.11:6789/0,c6z3zz1=157.249.160.106:6789/0}
Oct 29 14:29:45 95z3zz1 ceph-mon: -322> 2015-10-29 14:29:44.635420
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8d680 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -321> 2015-10-29 14:29:44.635463
7f315104e700 0 log_channel(cluster) log [INF] : pgmap v33661653: 21936 pgs:
21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail
Oct 29 14:29:45 95z3zz1 ceph-mon: -320> 2015-10-29 14:29:44.635466
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -319> 2015-10-29 14:29:44.635478
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -318> 2015-10-29 14:29:44.635479
7f315104e700 10 log_client log_queue is 5 last_log 5 sent 4 num 5 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -317> 2015-10-29 14:29:44.635481
7f315104e700 10 log_client will send 2015-10-29 14:29:44.635466 mon.0
157.249.160.11:6789/0 5 : cluster [INF] pgmap v33661653: 21936 pgs: 21936
active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail
Oct 29 14:29:45 95z3zz1 ceph-mon: -316> 2015-10-29 14:29:44.635486
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -315> 2015-10-29 14:29:44.635531
7f315104e700 0 log_channel(cluster) log [INF] : mdsmap e89: 1/1/1 up
{0=56z3zz1=up:active}
Oct 29 14:29:45 95z3zz1 ceph-mon: -314> 2015-10-29 14:29:44.635535
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -313> 2015-10-29 14:29:44.635546
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -312> 2015-10-29 14:29:44.635548
7f315104e700 10 log_client log_queue is 6 last_log 6 sent 5 num 6 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -311> 2015-10-29 14:29:44.635550
7f315104e700 10 log_client will send 2015-10-29 14:29:44.635534 mon.0
157.249.160.11:6789/0 6 : cluster [INF] mdsmap e89: 1/1/1 up
{0=56z3zz1=up:active}
Oct 29 14:29:45 95z3zz1 ceph-mon: -310> 2015-10-29 14:29:44.635555
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8dd40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -309> 2015-10-29 14:29:44.635656
7f315104e700 0 log_channel(cluster) log [INF] : osdmap e61385: 48 osds: 48 up,
48 in
Oct 29 14:29:45 95z3zz1 ceph-mon: -308> 2015-10-29 14:29:44.635659
7f315104e700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -307> 2015-10-29 14:29:44.635672
7f315104e700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -306> 2015-10-29 14:29:44.635673
7f315104e700 10 log_client log_queue is 7 last_log 7 sent 6 num 7 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -305> 2015-10-29 14:29:44.635675
7f315104e700 10 log_client will send 2015-10-29 14:29:44.635659 mon.0
157.249.160.11:6789/0 7 : cluster [INF] osdmap e61385: 48 osds: 48 up, 48 in
Oct 29 14:29:45 95z3zz1 ceph-mon: -304> 2015-10-29 14:29:44.635680
7f315104e700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8d440 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -303> 2015-10-29 14:29:44.635736
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635737 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -302> 2015-10-29 14:29:44.635755
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635756 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -301> 2015-10-29 14:29:44.635791
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.12:0/2056508 --
auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8e400 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -300> 2015-10-29 14:29:44.635813
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635814 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -299> 2015-10-29 14:29:44.635832
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8e1c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -298> 2015-10-29 14:29:44.635841
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635842 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -297> 2015-10-29 14:29:44.635854
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999375 ==== time_check( pong e 426 r 1 ts 2015-10-29 14:29:44.629005 ) v1
==== 36+0+0 (1025219847 0 0) 0x4f8cd80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -296> 2015-10-29 14:29:44.635874
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139019 ==== time_check( pong e 426 r 1 ts 2015-10-29 14:29:44.631212 ) v1
==== 36+0+0 (973715343 0 0) 0x4f8db00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -295> 2015-10-29 14:29:44.635890
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
time_check( report e 426 r 2 #skews 3 #latencies 3 ) v1 -- ?+0 0x4f8cd80
Oct 29 14:29:45 95z3zz1 ceph-mon: -294> 2015-10-29 14:29:44.635909
7f315104e700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
time_check( report e 426 r 2 #skews 3 #latencies 3 ) v1 -- ?+0 0x4f8c900
Oct 29 14:29:45 95z3zz1 ceph-mon: -293> 2015-10-29 14:29:44.635920
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d680 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -292> 2015-10-29 14:29:44.635926
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635927 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -291> 2015-10-29 14:29:44.635937
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999376 ==== paxos(lease_ack lc 62204013 fc 62203323 pn 0 opn 0) v3 ====
80+0+0 (1628804849 0 0) 0x5f96d00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -290> 2015-10-29 14:29:44.635948
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139020 ==== paxos(lease_ack lc 62204013 fc 62203323 pn 0 opn 0) v3 ====
80+0+0 (751612340 0 0) 0x5f66280 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -289> 2015-10-29 14:29:44.635958
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -288> 2015-10-29 14:29:44.635966
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635967 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -287> 2015-10-29 14:29:44.635977
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8dd40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -286> 2015-10-29 14:29:44.635983
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.635983 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -285> 2015-10-29 14:29:44.635994
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d440 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -284> 2015-10-29 14:29:44.635999
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636000 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -283> 2015-10-29 14:29:44.636009
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139001 ==== forward(pg_stats(3 pgs tid 19047 v 0) v1 caps allow profile osd
tid 554 con_features 37154696925806591) to leader v3 ==== 2041+0+0 (154590633 0
0) 0x4e12100 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -282> 2015-10-29 14:29:44.636029
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636029 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -281> 2015-10-29 14:29:44.636050
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636051 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -280> 2015-10-29 14:29:44.636078
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999351 ==== forward(pg_stats(6 pgs tid 939 v 0) v1 caps allow profile osd tid
454 con_features 37154696925806591) to leader v3 ==== 3552+0+0 (2093592133 0 0)
0x4e11e40 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -279> 2015-10-29 14:29:44.636089
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636090 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -278> 2015-10-29 14:29:44.636102
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636103 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -277> 2015-10-29 14:29:44.636119
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139002 ==== forward(pg_stats(3 pgs tid 19188 v 0) v1 caps allow profile osd
tid 555 con_features 37154696925806591) to leader v3 ==== 2041+0+0 (4008145650
0 0) 0x4e12ec0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -276> 2015-10-29 14:29:44.636130
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636131 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -275> 2015-10-29 14:29:44.636141
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636142 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -274> 2015-10-29 14:29:44.636154
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999352 ==== forward(pg_stats(5 pgs tid 940 v 0) v1 caps allow profile osd tid
455 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (29326344 0 0)
0x4e13440 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -273> 2015-10-29 14:29:44.636163
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636163 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -272> 2015-10-29 14:29:44.636175
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636176 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -271> 2015-10-29 14:29:44.636196
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139003 ==== forward(pg_stats(1 pgs tid 19164 v 0) v1 caps allow profile osd
tid 556 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (219839371 0
0) 0x4e13c80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -270> 2015-10-29 14:29:44.636207
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636208 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -269> 2015-10-29 14:29:44.636219
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636219 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -268> 2015-10-29 14:29:44.636231
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999353 ==== forward(pg_stats(3 pgs tid 19130 v 0) v1 caps allow profile osd
tid 456 con_features 37154696925806591) to leader v3 ==== 2049+0+0 (667527532 0
0) 0x5f4a580 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -267> 2015-10-29 14:29:44.636242
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636243 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -266> 2015-10-29 14:29:44.636254
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636255 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -265> 2015-10-29 14:29:44.636268
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139004 ==== forward(pg_stats(6 pgs tid 18939 v 0) v1 caps allow profile osd
tid 557 con_features 37154696925806591) to leader v3 ==== 3544+0+0 (4151017389
0 0) 0x4e13700 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -264> 2015-10-29 14:29:44.636278
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636279 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -263> 2015-10-29 14:29:44.636291
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636292 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -262> 2015-10-29 14:29:44.636308
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4878761
172.16.6.12:0/2056508 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0
(909137154 0 0) 0x4f8f600 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -261> 2015-10-29 14:29:44.636316
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636316 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -260> 2015-10-29 14:29:44.636439
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.12:0/2056508 --
auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8db00 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -259> 2015-10-29 14:29:44.636457
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999354 ==== forward(pg_stats(2 pgs tid 19158 v 0) v1 caps allow profile osd
tid 457 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (1302266018
0 0) 0x5f4a000 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -258> 2015-10-29 14:29:44.636470
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636470 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -257> 2015-10-29 14:29:44.636484
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636484 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -256> 2015-10-29 14:29:44.636507
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139005 ==== forward(pg_stats(7 pgs tid 18875 v 0) v1 caps allow profile osd
tid 558 con_features 37154696925806591) to leader v3 ==== 4045+0+0 (3611257738
0 0) 0x5f4adc0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -255> 2015-10-29 14:29:44.636534
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636535 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -254> 2015-10-29 14:29:44.636546
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636546 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -253> 2015-10-29 14:29:44.636562
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999355 ==== forward(pg_stats(2 pgs tid 943 v 0) v1 caps allow profile osd tid
458 con_features 37154696925806591) to leader v3 ==== 1544+0+0 (1619202949 0 0)
0x5f4b8c0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -252> 2015-10-29 14:29:44.636580
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636581 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -251> 2015-10-29 14:29:44.636593
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636594 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -250> 2015-10-29 14:29:44.636605
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139006 ==== forward(pg_stats(5 pgs tid 19001 v 0) v1 caps allow profile osd
tid 559 con_features 37154696925806591) to leader v3 ==== 3043+0+0 (744702348 0
0) 0x5f4a840 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -249> 2015-10-29 14:29:44.636615
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636616 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -248> 2015-10-29 14:29:44.636626
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636626 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -247> 2015-10-29 14:29:44.636642
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999356 ==== forward(pg_stats(1 pgs tid 19101 v 0) v1 caps allow profile osd
tid 459 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1036056958
0 0) 0x5f4b340 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -246> 2015-10-29 14:29:44.636652
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636652 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -245> 2015-10-29 14:29:44.636663
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636664 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -244> 2015-10-29 14:29:44.636674
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139007 ==== forward(pg_stats(1 pgs tid 18881 v 0) v1 caps allow profile osd
tid 560 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (3429570332
0 0) 0x5f4c680 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -243> 2015-10-29 14:29:44.636693
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636693 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -242> 2015-10-29 14:29:44.636705
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636705 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -241> 2015-10-29 14:29:44.636717
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999357 ==== forward(pg_stats(5 pgs tid 943 v 0) v1 caps allow profile osd tid
460 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (1567813846 0 0)
0x5f4d440 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -240> 2015-10-29 14:29:44.636726
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636726 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -239> 2015-10-29 14:29:44.636738
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636738 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -238> 2015-10-29 14:29:44.636753
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139008 ==== forward(pg_stats(11 pgs tid 19000 v 0) v1 caps allow profile
osd tid 561 con_features 37154696925806591) to leader v3 ==== 6049+0+0
(2318291987 0 0) 0x5f4c100 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -237> 2015-10-29 14:29:44.636763
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636763 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -236> 2015-10-29 14:29:44.636774
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636774 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -235> 2015-10-29 14:29:44.636801
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999358 ==== forward(pg_stats(2 pgs tid 19093 v 0) v1 caps allow profile osd
tid 461 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (3377069825
0 0) 0x5f4cec0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -234> 2015-10-29 14:29:44.636812
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636813 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -233> 2015-10-29 14:29:44.636885
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636886 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -232> 2015-10-29 14:29:44.636901
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139009 ==== forward(pg_stats(1 pgs tid 19226 v 0) v1 caps allow profile osd
tid 562 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1339636414
0 0) 0x5f4dc80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -231> 2015-10-29 14:29:44.636912
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636912 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -230> 2015-10-29 14:29:44.636924
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636924 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -229> 2015-10-29 14:29:44.636935
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999359 ==== forward(pg_stats(2 pgs tid 19095 v 0) v1 caps allow profile osd
tid 462 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (2108816789
0 0) 0x5f7cdc0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -228> 2015-10-29 14:29:44.636944
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636945 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -227> 2015-10-29 14:29:44.636955
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636956 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -226> 2015-10-29 14:29:44.636968
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139010 ==== forward(pg_stats(9 pgs tid 19023 v 0) v1 caps allow profile osd
tid 563 con_features 37154696925806591) to leader v3 ==== 5047+0+0 (4205152908
0 0) 0x5f7e100 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -225> 2015-10-29 14:29:44.636978
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636979 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -224> 2015-10-29 14:29:44.636989
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.636990 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -223> 2015-10-29 14:29:44.637010
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999360 ==== forward(pg_stats(8 pgs tid 19052 v 0) v1 caps allow profile osd
tid 463 con_features 37154696925806591) to leader v3 ==== 4546+0+0 (1173158927
0 0) 0x5f7c840 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -222> 2015-10-29 14:29:44.637020
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637021 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -221> 2015-10-29 14:29:44.637031
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637031 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -220> 2015-10-29 14:29:44.637056
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139011 ==== forward(pg_stats(2 pgs tid 19096 v 0) v1 caps allow profile osd
tid 564 con_features 37154696925806591) to leader v3 ==== 1540+0+0 (3933631993
0 0) 0x5f7db80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -219> 2015-10-29 14:29:44.637067
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637067 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -218> 2015-10-29 14:29:44.637079
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637079 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -217> 2015-10-29 14:29:44.637090
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4878761
172.16.6.12:0/2056508 3 ==== mon_subscribe({monmap=8+,osdmap=61386}) v2 ====
42+0+0 (2169229493 0 0) 0x6028000 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -216> 2015-10-29 14:29:44.637110
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.12:0/2056508 --
mon_subscribe_ack(300s) v1 -- ?+0 0x4f303c0 con 0x4d547e0
Oct 29 14:29:45 95z3zz1 ceph-mon: -215> 2015-10-29 14:29:44.637123
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999361 ==== forward(pg_stats(6 pgs tid 939 v 0) v1 caps allow profile osd tid
464 con_features 37154696925806591) to leader v3 ==== 3552+0+0 (1387593706 0 0)
0x5f7c2c0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -214> 2015-10-29 14:29:44.637132
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637133 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -213> 2015-10-29 14:29:44.637144
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637145 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -212> 2015-10-29 14:29:44.637160
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139012 ==== forward(pg_stats(1 pgs tid 19135 v 0) v1 caps allow profile osd
tid 565 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (534192359 0
0) 0x5f7d600 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -211> 2015-10-29 14:29:44.637170
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637170 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -210> 2015-10-29 14:29:44.637181
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637182 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -209> 2015-10-29 14:29:44.637192
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999362 ==== forward(pg_stats(1 pgs tid 18922 v 0) v1 caps allow profile osd
tid 465 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (779363514 0
0) 0x5f7fc80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -208> 2015-10-29 14:29:44.637202
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637202 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -207> 2015-10-29 14:29:44.637213
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637214 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -206> 2015-10-29 14:29:44.637224
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139013 ==== forward(pg_stats(3 pgs tid 18948 v 0) v1 caps allow profile osd
tid 566 con_features 37154696925806591) to leader v3 ==== 2041+0+0 (2346298246
0 0) 0x5f7d080 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -205> 2015-10-29 14:29:44.637234
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637234 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -204> 2015-10-29 14:29:44.637245
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637245 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -203> 2015-10-29 14:29:44.637257
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999363 ==== forward(pg_stats(5 pgs tid 18985 v 0) v1 caps allow profile osd
tid 466 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (399880181 0
0) 0x5f89340 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -202> 2015-10-29 14:29:44.637266
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637266 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -201> 2015-10-29 14:29:44.637277
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637277 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -200> 2015-10-29 14:29:44.637293
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139014 ==== forward(pg_stats(1 pgs tid 18994 v 0) v1 caps allow profile osd
tid 567 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1768182538
0 0) 0x5f7f440 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -199> 2015-10-29 14:29:44.637302
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637303 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -198> 2015-10-29 14:29:44.637314
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637314 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -197> 2015-10-29 14:29:44.637324
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999364 ==== forward(pg_stats(2 pgs tid 942 v 0) v1 caps allow profile osd tid
467 con_features 37154696925806591) to leader v3 ==== 1544+0+0 (1147565120 0 0)
0x5f88dc0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -196> 2015-10-29 14:29:44.637333
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637334 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -195> 2015-10-29 14:29:44.637344
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637345 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -194> 2015-10-29 14:29:44.637365
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139015 ==== forward(pg_stats(1 pgs tid 18954 v 0) v1 caps allow profile osd
tid 568 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (1581327975
0 0) 0x5f7eec0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -193> 2015-10-29 14:29:44.637376
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637376 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -192> 2015-10-29 14:29:44.637388
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637388 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -191> 2015-10-29 14:29:44.637398
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999365 ==== forward(pg_stats(9 pgs tid 18943 v 0) v1 caps allow profile osd
tid 468 con_features 37154696925806591) to leader v3 ==== 5051+0+0 (64146959 0
0) 0x5f88840 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -190> 2015-10-29 14:29:44.637407
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637408 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -189> 2015-10-29 14:29:44.637427
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637428 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -188> 2015-10-29 14:29:44.637445
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139016 ==== forward(pg_stats(1 pgs tid 19179 v 0) v1 caps allow profile osd
tid 569 con_features 37154696925806591) to leader v3 ==== 1039+0+0 (4190925099
0 0) 0x5f7e940 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -187> 2015-10-29 14:29:44.637455
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637456 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -186> 2015-10-29 14:29:44.637467
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637467 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -185> 2015-10-29 14:29:44.637477
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999366 ==== forward(pg_stats(3 pgs tid 19165 v 0) v1 caps allow profile osd
tid 469 con_features 37154696925806591) to leader v3 ==== 2069+0+0 (2844274865
0 0) 0x5f882c0 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -184> 2015-10-29 14:29:44.637495
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637496 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -183> 2015-10-29 14:29:44.637507
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637507 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -182> 2015-10-29 14:29:44.637519
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139017 ==== forward(pg_stats(4 pgs tid 18988 v 0) v1 caps allow profile osd
tid 570 con_features 37154696925806591) to leader v3 ==== 2546+0+0 (1830822204
0 0) 0x5f7e3c0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -181> 2015-10-29 14:29:44.637529
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637529 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -180> 2015-10-29 14:29:44.637541
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637541 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -179> 2015-10-29 14:29:44.637554
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999367 ==== forward(pg_stats(6 pgs tid 19221 v 0) v1 caps allow profile osd
tid 470 con_features 37154696925806591) to leader v3 ==== 3544+0+0 (2898347798
0 0) 0x5f8ac00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -178> 2015-10-29 14:29:44.637563
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637573 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -177> 2015-10-29 14:29:44.637585
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637585 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -176> 2015-10-29 14:29:44.637618
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999368 ==== forward(pg_stats(11 pgs tid 945 v 0) v1 caps allow profile osd
tid 471 con_features 37154696925806591) to leader v3 ==== 6053+0+0 (2902556831
0 0) 0x5f8a680 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -175> 2015-10-29 14:29:44.637628
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637629 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -174> 2015-10-29 14:29:44.637640
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637640 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -173> 2015-10-29 14:29:44.637658
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999369 ==== forward(pg_stats(2 pgs tid 19072 v 0) v1 caps allow profile osd
tid 472 con_features 37154696925806591) to leader v3 ==== 1548+0+0 (3189990860
0 0) 0x5f8a100 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -172> 2015-10-29 14:29:44.637667
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637668 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -171> 2015-10-29 14:29:44.637678
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637679 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -170> 2015-10-29 14:29:44.637690
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999370 ==== forward(pg_stats(2 pgs tid 943 v 0) v1 caps allow profile osd tid
473 con_features 37154696925806591) to leader v3 ==== 1544+0+0 (2273230390 0 0)
0x5f89b80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -169> 2015-10-29 14:29:44.637699
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637700 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -168> 2015-10-29 14:29:44.637710
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637711 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -167> 2015-10-29 14:29:44.637721
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999371 ==== forward(pg_stats(5 pgs tid 945 v 0) v1 caps allow profile osd tid
474 con_features 37154696925806591) to leader v3 ==== 3051+0+0 (125037623 0 0)
0x5f89600 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -166> 2015-10-29 14:29:44.637739
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637740 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -165> 2015-10-29 14:29:44.637751
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637752 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -164> 2015-10-29 14:29:44.637765
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999372 ==== forward(pg_stats(7 pgs tid 19160 v 0) v1 caps allow profile osd
tid 475 con_features 37154696925806591) to leader v3 ==== 4045+0+0 (1236860648
0 0) 0x5f9d600 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -163> 2015-10-29 14:29:44.637774
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637775 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -162> 2015-10-29 14:29:44.637786
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637786 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -161> 2015-10-29 14:29:44.637811
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999373 ==== forward(pg_stats(2 pgs tid 941 v 0) v1 caps allow profile osd tid
476 con_features 37154696925806591) to leader v3 ==== 1548+0+0 (4082189338 0 0)
0x5f9d080 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -160> 2015-10-29 14:29:44.637821
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637822 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -159> 2015-10-29 14:29:44.637832
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.637833 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -158> 2015-10-29 14:29:44.678540
7f314ea44700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ef1000 sd=13 :6789 s=0
pgs=0 cs=0 l=0 c=0x603a940).accept sd=13 172.16.6.5:42554/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -157> 2015-10-29 14:29:44.679435
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4399776
172.16.6.5:0/2080099 1 ==== auth(proto 0 35 bytes epoch 7) v1 ==== 65+0+0
(1271301283 0 0) 0x4f8e400 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -156> 2015-10-29 14:29:44.679459
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.679461 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -155> 2015-10-29 14:29:44.679495
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.5:0/2080099 --
auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8f600 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -154> 2015-10-29 14:29:44.680046
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4399776
172.16.6.5:0/2080099 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0
(3085145404 0 0) 0x4f8f840 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -153> 2015-10-29 14:29:44.680064
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.680064 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -152> 2015-10-29 14:29:44.680210
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.5:0/2080099 --
auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8e400 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -151> 2015-10-29 14:29:44.680803
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.4399776
172.16.6.5:0/2080099 3 ==== mon_subscribe({monmap=8+,osdmap=61386}) v2 ====
42+0+0 (2169229493 0 0) 0x602b200 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -150> 2015-10-29 14:29:44.680827
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.5:0/2080099 --
mon_subscribe_ack(300s) v1 -- ?+0 0x4f30000 con 0x603a940
Oct 29 14:29:45 95z3zz1 ceph-mon: -149> 2015-10-29 14:29:44.684170
7f314e842700 1 -- 157.249.160.11:6789/0 >> :/0 pipe(0x5ee2000 sd=14 :6789 s=0
pgs=0 cs=0 l=0 c=0x603a680).accept sd=14 172.16.6.4:51408/0
Oct 29 14:29:45 95z3zz1 ceph-mon: -148> 2015-10-29 14:29:44.684773
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.1638129
172.16.6.4:0/1090975 1 ==== auth(proto 0 35 bytes epoch 7) v1 ==== 65+0+0
(2584606190 0 0) 0x4f8f600 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -147> 2015-10-29 14:29:44.684790
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.684791 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -146> 2015-10-29 14:29:44.684825
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.4:0/1090975 --
auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8f840 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -145> 2015-10-29 14:29:44.685241
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999377 ==== forward(log(1 entries) v1 caps allow * tid 477 con_features 0) to
leader v3 ==== 584+0+0 (1309431426 0 0) 0x5f9cb00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -144> 2015-10-29 14:29:44.685263
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.685264 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -143> 2015-10-29 14:29:44.685325
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.1638129
172.16.6.4:0/1090975 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0
(3893551484 0 0) 0x4f8fa80 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -142> 2015-10-29 14:29:44.685336
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
is_readable = 1 - now=2015-10-29 14:29:44.685337 lease_expire=2015-10-29
14:29:49.635042 has v0 lc 62204013
Oct 29 14:29:45 95z3zz1 ceph-mon: -141> 2015-10-29 14:29:44.685459
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.4:0/1090975 --
auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x4f8f600 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -140> 2015-10-29 14:29:44.685938
7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204013)
queue_pending_finisher 0x5f0aaf0
Oct 29 14:29:45 95z3zz1 ceph-mon: -139> 2015-10-29 14:29:44.686171
7f315104e700 1 -- 157.249.160.11:6789/0 <== client.1638129
172.16.6.4:0/1090975 3 ==== mon_subscribe({monmap=8+,osdmap=61386}) v2 ====
42+0+0 (2169229493 0 0) 0x602b400 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -138> 2015-10-29 14:29:44.686376
7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(begin lc 62204013 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f200
Oct 29 14:29:45 95z3zz1 ceph-mon: -137> 2015-10-29 14:29:44.686400
7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(begin lc 62204013 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f700
Oct 29 14:29:45 95z3zz1 ceph-mon: -136> 2015-10-29 14:29:44.686434
7f315104e700 1 -- 157.249.160.11:6789/0 --> 172.16.6.4:0/1090975 --
mon_subscribe_ack(300s) v1 -- ?+0 0x4f305a0 con 0x603a680
Oct 29 14:29:45 95z3zz1 ceph-mon: -135> 2015-10-29 14:29:44.687223
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999378 ==== paxos(accept lc 62204013 fc 0 pn 21100 opn 0) v3 ==== 80+0+0
(1988427842 0 0) 0x604f200 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -134> 2015-10-29 14:29:44.687599
7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos updating c
62203323..62204013) queue_pending_finisher 0x5f080b0
Oct 29 14:29:45 95z3zz1 ceph-mon: -133> 2015-10-29 14:29:44.687662
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139021 ==== paxos(accept lc 62204013 fc 0 pn 21100 opn 0) v3 ==== 80+0+0
(1732951130 0 0) 0x604f700 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -132> 2015-10-29 14:29:44.688677
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(commit lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f980
Oct 29 14:29:45 95z3zz1 ceph-mon: -131> 2015-10-29 14:29:44.688697
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(commit lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604fc00
Oct 29 14:29:45 95z3zz1 ceph-mon: -130> 2015-10-29 14:29:44.689250
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(lease lc 62204014 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60ea500
Oct 29 14:29:45 95z3zz1 ceph-mon: -129> 2015-10-29 14:29:44.689262
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(lease lc 62204014 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60ea280
Oct 29 14:29:45 95z3zz1 ceph-mon: -128> 2015-10-29 14:29:44.689297
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
1) v1 -- ?+0 0x4f30780 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -127> 2015-10-29 14:29:44.689320
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
2) v1 -- ?+0 0x4f30b40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -126> 2015-10-29 14:29:44.689326
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
3) v1 -- ?+0 0x4f30960 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -125> 2015-10-29 14:29:44.689335
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
4) v1 -- ?+0 0x4f310e0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -124> 2015-10-29 14:29:44.689343
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
5) v1 -- ?+0 0x4f30f00 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -123> 2015-10-29 14:29:44.689336
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 1) v1 ==== 0+0+0 (0 0 0) 0x4f30780 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -122> 2015-10-29 14:29:44.689357
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
6) v1 -- ?+0 0x4f30d20 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -121> 2015-10-29 14:29:44.689366
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(last
7) v1 -- ?+0 0x4f31860 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -120> 2015-10-29 14:29:44.689374
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(log(last 454) v1 tid 477) v2 -- ?+0 0x60ea000 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -119> 2015-10-29 14:29:44.690195
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(begin lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f96a80
Oct 29 14:29:45 95z3zz1 ceph-mon: -118> 2015-10-29 14:29:44.690229
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(begin lc 62204014 fc 0 pn 21100 opn 0) v3 -- ?+0 0x60eaf00
Oct 29 14:29:45 95z3zz1 ceph-mon: -117> 2015-10-29 14:29:44.690371
7f315104e700 10 log_client handle_log_ack log(last 1) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -116> 2015-10-29 14:29:44.690380
7f315104e700 10 log_client logged 2015-10-29 14:29:44.629699 mon.0
157.249.160.11:6789/0 1 : cluster [INF] mon.95z3zz1 calling new monitor election
Oct 29 14:29:45 95z3zz1 ceph-mon: -115> 2015-10-29 14:29:44.690395
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 2) v1 ==== 0+0+0 (0 0 0) 0x4f30b40 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -114> 2015-10-29 14:29:44.690402
7f315104e700 10 log_client handle_log_ack log(last 2) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -113> 2015-10-29 14:29:44.690403
7f315104e700 10 log_client logged 2015-10-29 14:29:44.631552 mon.0
157.249.160.11:6789/0 2 : cluster [INF] mon.95z3zz1@0 won leader election with
quorum 0,1,2
Oct 29 14:29:45 95z3zz1 ceph-mon: -112> 2015-10-29 14:29:44.690410
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999379 ==== paxos(lease_ack lc 62204014 fc 62203323 pn 0 opn 0) v3 ====
80+0+0 (257771121 0 0) 0x60ea000 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -111> 2015-10-29 14:29:44.690425
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139022 ==== paxos(lease_ack lc 62204014 fc 62203323 pn 0 opn 0) v3 ====
80+0+0 (3143862888 0 0) 0x60ea280 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -110> 2015-10-29 14:29:44.690434
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 3) v1 ==== 0+0+0 (0 0 0) 0x4f30960 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -109> 2015-10-29 14:29:44.690439
7f315104e700 10 log_client handle_log_ack log(last 3) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -108> 2015-10-29 14:29:44.690440
7f315104e700 10 log_client logged 2015-10-29 14:29:44.634903 mon.0
157.249.160.11:6789/0 3 : cluster [INF] HEALTH_WARN; noout flag(s) set
Oct 29 14:29:45 95z3zz1 ceph-mon: -107> 2015-10-29 14:29:44.690448
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 4) v1 ==== 0+0+0 (0 0 0) 0x4f310e0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -106> 2015-10-29 14:29:44.690462
7f315104e700 10 log_client handle_log_ack log(last 4) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -105> 2015-10-29 14:29:44.690463
7f315104e700 10 log_client logged 2015-10-29 14:29:44.635376 mon.0
157.249.160.11:6789/0 4 : cluster [INF] monmap e7: 3 mons at
{56z3zz1=157.249.160.107:6789/0,95z3zz1=157.249.160.11:6789/0,c6z3zz1=157.249.160.106:6789/0}
Oct 29 14:29:45 95z3zz1 ceph-mon: -104> 2015-10-29 14:29:44.690470
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 5) v1 ==== 0+0+0 (0 0 0) 0x4f30f00 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -103> 2015-10-29 14:29:44.690475
7f315104e700 10 log_client handle_log_ack log(last 5) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -102> 2015-10-29 14:29:44.690489
7f315104e700 10 log_client logged 2015-10-29 14:29:44.635466 mon.0
157.249.160.11:6789/0 5 : cluster [INF] pgmap v33661653: 21936 pgs: 21936
active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail
Oct 29 14:29:45 95z3zz1 ceph-mon: -101> 2015-10-29 14:29:44.690496
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 6) v1 ==== 0+0+0 (0 0 0) 0x4f30d20 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -100> 2015-10-29 14:29:44.690502
7f315104e700 10 log_client handle_log_ack log(last 6) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -99> 2015-10-29 14:29:44.690503
7f315104e700 10 log_client logged 2015-10-29 14:29:44.635534 mon.0
157.249.160.11:6789/0 6 : cluster [INF] mdsmap e89: 1/1/1 up
{0=56z3zz1=up:active}
Oct 29 14:29:45 95z3zz1 ceph-mon: -98> 2015-10-29 14:29:44.690509
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(last 7) v1 ==== 0+0+0 (0 0 0) 0x4f31860 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -97> 2015-10-29 14:29:44.690515
7f315104e700 10 log_client handle_log_ack log(last 7) v1
Oct 29 14:29:45 95z3zz1 ceph-mon: -96> 2015-10-29 14:29:44.690516
7f315104e700 10 log_client logged 2015-10-29 14:29:44.635659 mon.0
157.249.160.11:6789/0 7 : cluster [INF] osdmap e61385: 48 osds: 48 up, 48 in
Oct 29 14:29:45 95z3zz1 ceph-mon: -95> 2015-10-29 14:29:44.692160
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999380 ==== paxos(accept lc 62204014 fc 0 pn 21100 opn 0) v3 ==== 80+0+0
(2150592006 0 0) 0x5f96a80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -94> 2015-10-29 14:29:44.692215
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139023 ==== paxos(accept lc 62204014 fc 0 pn 21100 opn 0) v3 ==== 80+0+0
(4160712019 0 0) 0x60eaf00 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -93> 2015-10-29 14:29:44.693301
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(commit lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x60eaa00
Oct 29 14:29:45 95z3zz1 ceph-mon: -92> 2015-10-29 14:29:44.693340
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(commit lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x60ea780
Oct 29 14:29:45 95z3zz1 ceph-mon: -91> 2015-10-29 14:29:44.696728
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(lease lc 62204015 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60eb900
Oct 29 14:29:45 95z3zz1 ceph-mon: -90> 2015-10-29 14:29:44.696756
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(lease lc 62204015 fc 62203323 pn 0 opn 0) v3 -- ?+0 0x60eb680
Oct 29 14:29:45 95z3zz1 ceph-mon: -89> 2015-10-29 14:29:44.696813
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(3 pgs tid 19047) v1 tid 554) v2 -- ?+0 0x5f67b80 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -88> 2015-10-29 14:29:44.696850
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(6 pgs tid 939) v1 tid 454) v2 -- ?+0 0x4dbaa00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -87> 2015-10-29 14:29:44.696866
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(3 pgs tid 19188) v1 tid 555) v2 -- ?+0 0x4dbac80 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -86> 2015-10-29 14:29:44.696879
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(5 pgs tid 940) v1 tid 455) v2 -- ?+0 0x5f68800 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -85> 2015-10-29 14:29:44.696911
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 19164) v1 tid 556) v2 -- ?+0 0x4dbcf80 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -84> 2015-10-29 14:29:44.696929
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(3 pgs tid 19130) v1 tid 456) v2 -- ?+0 0x4dbd980 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -83> 2015-10-29 14:29:44.696942
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(6 pgs tid 18939) v1 tid 557) v2 -- ?+0 0x5f74c80 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -82> 2015-10-29 14:29:44.696956
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 19158) v1 tid 457) v2 -- ?+0 0x4dbd700 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -81> 2015-10-29 14:29:44.696967
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(7 pgs tid 18875) v1 tid 558) v2 -- ?+0 0x5f74500 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -80> 2015-10-29 14:29:44.696980
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 943) v1 tid 458) v2 -- ?+0 0x5f70280 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -79> 2015-10-29 14:29:44.697009
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(5 pgs tid 19001) v1 tid 559) v2 -- ?+0 0x5f74000 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -78> 2015-10-29 14:29:44.697024
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(1 pgs tid 19101) v1 tid 459) v2 -- ?+0 0x5f74f00 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -77> 2015-10-29 14:29:44.697035
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 18881) v1 tid 560) v2 -- ?+0 0x5f76a80 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -76> 2015-10-29 14:29:44.697059
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(5 pgs tid 943) v1 tid 460) v2 -- ?+0 0x5f77480 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -75> 2015-10-29 14:29:44.697073
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(11 pgs tid 19000) v1 tid 561) v2 -- ?+0 0x5f76800 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -74> 2015-10-29 14:29:44.697089
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 19093) v1 tid 461) v2 -- ?+0 0x5f77200 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -73> 2015-10-29 14:29:44.697102
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 19226) v1 tid 562) v2 -- ?+0 0x5f73c00 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -72> 2015-10-29 14:29:44.697123
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 19095) v1 tid 462) v2 -- ?+0 0x5f73480 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -71> 2015-10-29 14:29:44.697136
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(9 pgs tid 19023) v1 tid 563) v2 -- ?+0 0x5f81400 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -70> 2015-10-29 14:29:44.697149
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(8 pgs tid 19052) v1 tid 463) v2 -- ?+0 0x5f70a00 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -69> 2015-10-29 14:29:44.697161
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(2 pgs tid 19096) v1 tid 564) v2 -- ?+0 0x5f80f00 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -68> 2015-10-29 14:29:44.697183
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(6 pgs tid 939) v1 tid 464) v2 -- ?+0 0x5f82800 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -67> 2015-10-29 14:29:44.697197
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 19135) v1 tid 565) v2 -- ?+0 0x5f83980 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -66> 2015-10-29 14:29:44.697216
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(1 pgs tid 18922) v1 tid 465) v2 -- ?+0 0x5f82300 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -65> 2015-10-29 14:29:44.697228
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(3 pgs tid 18948) v1 tid 566) v2 -- ?+0 0x5f85680 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -64> 2015-10-29 14:29:44.697250
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(5 pgs tid 18985) v1 tid 466) v2 -- ?+0 0x5f82080 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -63> 2015-10-29 14:29:44.697265
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 18994) v1 tid 567) v2 -- ?+0 0x5f85400 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -62> 2015-10-29 14:29:44.697279
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 942) v1 tid 467) v2 -- ?+0 0x5f81900 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -61> 2015-10-29 14:29:44.697303
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 18954) v1 tid 568) v2 -- ?+0 0x5f84780 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -60> 2015-10-29 14:29:44.697319
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(9 pgs tid 18943) v1 tid 468) v2 -- ?+0 0x5f81680 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -59> 2015-10-29 14:29:44.697337
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(1 pgs tid 19179) v1 tid 569) v2 -- ?+0 0x5f81e00 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -58> 2015-10-29 14:29:44.697348
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(3 pgs tid 19165) v1 tid 469) v2 -- ?+0 0x5f87c00 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -57> 2015-10-29 14:29:44.697359
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.107:6789/0 --
route(pg_stats_ack(4 pgs tid 18988) v1 tid 570) v2 -- ?+0 0x5f81b80 con
0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -56> 2015-10-29 14:29:44.697371
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(6 pgs tid 19221) v1 tid 470) v2 -- ?+0 0x5f87980 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -55> 2015-10-29 14:29:44.697391
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(11 pgs tid 945) v1 tid 471) v2 -- ?+0 0x5f85e00 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -54> 2015-10-29 14:29:44.697418
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 19072) v1 tid 472) v2 -- ?+0 0x5f91400 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -53> 2015-10-29 14:29:44.697470
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 943) v1 tid 473) v2 -- ?+0 0x5f95b80 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -52> 2015-10-29 14:29:44.697511
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(5 pgs tid 945) v1 tid 474) v2 -- ?+0 0x5f95680 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -51> 2015-10-29 14:29:44.697565
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(7 pgs tid 19160) v1 tid 475) v2 -- ?+0 0x5f95180 con
0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -50> 2015-10-29 14:29:44.697606
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.106:6789/0 --
route(pg_stats_ack(2 pgs tid 941) v1 tid 476) v2 -- ?+0 0x5f94500 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -49> 2015-10-29 14:29:44.697678
7f3154801700 0 log_channel(cluster) log [INF] : pgmap v33661654: 21936 pgs:
21936 active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail; 6259
kB/s rd, 39388 kB/s wr, 4361 op/s
Oct 29 14:29:45 95z3zz1 ceph-mon: -48> 2015-10-29 14:29:44.697683
7f3154801700 0 log_channel(cluster) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -47> 2015-10-29 14:29:44.697707
7f3154801700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -46> 2015-10-29 14:29:44.697709
7f3154801700 10 log_client log_queue is 1 last_log 8 sent 7 num 1 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -45> 2015-10-29 14:29:44.697711
7f3154801700 10 log_client will send 2015-10-29 14:29:44.697682 mon.0
157.249.160.11:6789/0 8 : cluster [INF] pgmap v33661654: 21936 pgs: 21936
active+clean; 9824 GB data, 29683 GB used, 144 TB / 173 TB avail; 6259 kB/s rd,
39388 kB/s wr, 4361 op/s
Oct 29 14:29:45 95z3zz1 ceph-mon: -44> 2015-10-29 14:29:44.697716
7f3154801700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -43> 2015-10-29 14:29:44.697774
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8df80 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -42> 2015-10-29 14:29:44.697790
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:44.697791 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -41> 2015-10-29 14:29:44.698516
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999381 ==== paxos(lease_ack lc 62204015 fc 62203323 pn 0 opn 0) v3 ====
80+0+0 (2218565783 0 0) 0x5f94500 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -40> 2015-10-29 14:29:44.698776
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139024 ==== paxos(lease_ack lc 62204015 fc 62203323 pn 0 opn 0) v3 ====
80+0+0 (2124060710 0 0) 0x5f81b80 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -39> 2015-10-29 14:29:44.907157
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139025 ==== forward(log(1 entries) v1 caps allow * tid 571 con_features 0)
to leader v3 ==== 584+0+0 (4179428545 0 0) 0x5f8b9c0 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -38> 2015-10-29 14:29:44.907187
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:44.907188 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -37> 2015-10-29 14:29:45.074350
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999382 ==== forward(log(1 entries) v1 caps allow * tid 478 con_features 0) to
leader v3 ==== 584+0+0 (48471394 0 0) 0x5f9c840 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -36> 2015-10-29 14:29:45.074382
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.074384 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -35> 2015-10-29 14:29:45.303886
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999383 ==== forward(pg_stats(1 pgs tid 19053 v 0) v1 caps allow profile osd
tid 479 con_features 37154696925806591) to leader v3 ==== 1041+0+0 (2232892023
0 0) 0x5f9c580 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -34> 2015-10-29 14:29:45.303921
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.303922 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -33> 2015-10-29 14:29:45.303951
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.303952 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -32> 2015-10-29 14:29:45.374985
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139026 ==== forward(pg_stats(2 pgs tid 18876 v 0) v1 caps allow profile osd
tid 572 con_features 37154696925806591) to leader v3 ==== 1542+0+0 (1698272303
0 0) 0x5f8b700 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -31> 2015-10-29 14:29:45.375022
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.375023 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -30> 2015-10-29 14:29:45.375046
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.375046 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -29> 2015-10-29 14:29:45.591682
7f3155ddf700 5 asok(0x4daa000) AdminSocket: request 'get_command_descriptions'
'' to 0x4d32140 returned 2098 bytes
Oct 29 14:29:45 95z3zz1 ceph-mon: -28> 2015-10-29 14:29:45.593975
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139027 ==== forward(pg_stats(2 pgs tid 19024 v 0) v1 caps allow profile osd
tid 573 con_features 37154696925806591) to leader v3 ==== 1542+0+0 (2559589014
0 0) 0x5f8b180 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -27> 2015-10-29 14:29:45.594011
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.594012 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -26> 2015-10-29 14:29:45.594038
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.594039 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -25> 2015-10-29 14:29:45.596499
7f3155ddf700 0 log_channel(audit) log [DBG] : from='admin socket'
entity='admin socket' cmd='mon_status' args=[]: dispatch
Oct 29 14:29:45 95z3zz1 ceph-mon: -24> 2015-10-29 14:29:45.596508
7f3155ddf700 0 log_channel(audit) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -23> 2015-10-29 14:29:45.596513
7f3155ddf700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -22> 2015-10-29 14:29:45.596514
7f3155ddf700 10 log_client log_queue is 2 last_log 9 sent 8 num 2 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -21> 2015-10-29 14:29:45.596517
7f3155ddf700 10 log_client will send 2015-10-29 14:29:45.596508 mon.0
157.249.160.11:6789/0 9 : audit [DBG] from='admin socket' entity='admin socket'
cmd='mon_status' args=[]: dispatch
Oct 29 14:29:45 95z3zz1 ceph-mon: -20> 2015-10-29 14:29:45.596545
7f3155ddf700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x545c480 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -19> 2015-10-29 14:29:45.596600
7f3155ddf700 0 log_channel(audit) log [DBG] : from='admin socket'
entity='admin socket' cmd=mon_status args=[]: finished
Oct 29 14:29:45 95z3zz1 ceph-mon: -18> 2015-10-29 14:29:45.596603
7f3155ddf700 0 log_channel(audit) do_log log to syslog
Oct 29 14:29:45 95z3zz1 ceph-mon: -17> 2015-10-29 14:29:45.596604
7f3155ddf700 10 log_client _send_to_monlog to self
Oct 29 14:29:45 95z3zz1 ceph-mon: -16> 2015-10-29 14:29:45.596605
7f3155ddf700 10 log_client log_queue is 3 last_log 10 sent 9 num 3 unsent 1
sending 1
Oct 29 14:29:45 95z3zz1 ceph-mon: -15> 2015-10-29 14:29:45.596607
7f3155ddf700 10 log_client will send 2015-10-29 14:29:45.596603 mon.0
157.249.160.11:6789/0 10 : audit [DBG] from='admin socket' entity='admin
socket' cmd=mon_status args=[]: finished
Oct 29 14:29:45 95z3zz1 ceph-mon: -14> 2015-10-29 14:29:45.596613
7f3155ddf700 1 -- 157.249.160.11:6789/0 --> 157.249.160.11:6789/0 -- log(1
entries) v1 -- ?+0 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -13> 2015-10-29 14:29:45.596626
7f3155ddf700 5 asok(0x4daa000) AdminSocket: request 'mon_status' '' to
0x5e1dbe0 returned 855 bytes
Oct 29 14:29:45 95z3zz1 ceph-mon: -12> 2015-10-29 14:29:45.596652
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x545c480 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -11> 2015-10-29 14:29:45.596670
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.596672 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -10> 2015-10-29 14:29:45.596706
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.0 157.249.160.11:6789/0 0 ====
log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x4f8d8c0 con 0x4d53760
Oct 29 14:29:45 95z3zz1 ceph-mon: -9> 2015-10-29 14:29:45.596711
7f315104e700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
is_readable = 1 - now=2015-10-29 14:29:45.596712 lease_expire=2015-10-29
14:29:49.696726 has v0 lc 62204015
Oct 29 14:29:45 95z3zz1 ceph-mon: -8> 2015-10-29 14:29:45.693414
7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos active c 62203323..62204015)
queue_pending_finisher 0x5f096b0
Oct 29 14:29:45 95z3zz1 ceph-mon: -7> 2015-10-29 14:29:45.693887
7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(begin lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x604f480
Oct 29 14:29:45 95z3zz1 ceph-mon: -6> 2015-10-29 14:29:45.693918
7f315184f700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(begin lc 62204015 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f83200
Oct 29 14:29:45 95z3zz1 ceph-mon: -5> 2015-10-29 14:29:45.694012
7f315184f700 5 mon.95z3zz1@0(leader).paxos(paxos updating c
62203323..62204015) queue_pending_finisher 0x5f0aaf0
Oct 29 14:29:45 95z3zz1 ceph-mon: -4> 2015-10-29 14:29:45.694889
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.2 157.249.160.107:6789/0
1506139028 ==== paxos(accept lc 62204015 fc 0 pn 21100 opn 0) v3 ==== 80+0+0
(3498373181 0 0) 0x5f83200 con 0x4d53b80
Oct 29 14:29:45 95z3zz1 ceph-mon: -3> 2015-10-29 14:29:45.694925
7f315104e700 1 -- 157.249.160.11:6789/0 <== mon.1 157.249.160.106:6789/0
13999384 ==== paxos(accept lc 62204015 fc 0 pn 21100 opn 0) v3 ==== 80+0+0
(914064508 0 0) 0x604f480 con 0x4d53e40
Oct 29 14:29:45 95z3zz1 ceph-mon: -2> 2015-10-29 14:29:45.695372
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.1 157.249.160.106:6789/0 --
paxos(commit lc 62204016 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f97200
Oct 29 14:29:45 95z3zz1 ceph-mon: -1> 2015-10-29 14:29:45.695392
7f3154801700 1 -- 157.249.160.11:6789/0 --> mon.2 157.249.160.107:6789/0 --
paxos(commit lc 62204016 fc 0 pn 21100 opn 0) v3 -- ?+0 0x5f96f80
Oct 29 14:29:45 95z3zz1 ceph-mon: 0> 2015-10-29 14:29:45.697177
7f3154801700 -1 *** Caught signal (Segmentation fault) **#012 in thread
7f3154801700#012#012 ceph version 0.94.5
(9764da52395923e0b32908d83a9f7304401fee43)#012 1: /usr/bin/ceph-mon()
[0x9adefa]#012 2: (()+0x10340) [0x7f3159b63340]#012 3:
(std::_Rb_tree<std::string, std::pair<std::string const, std::string>,
std::_Select1st<std::pair<std::string const, std::string> >,
std::less<std::string>, std::allocator<std::pair<std::string const,
std::string> > >::find(std::string const&) const+0x25) [0x6518e5]#012 4:
(get_str_map_key(std::map<std::string, std::string, std::less<std::string>,
std::allocator<std::pair<std::string const, std::string> > > const&,
std::string const&, std::string const*)+0x1e) [0x8a002e]#012 5:
(LogMonitor::update_from_paxos(bool*)+0x87a) [0x6b0a5a]#012 6:
(PaxosService::refresh(bool*)+0x19a) [0x60432a]#012 7:
(Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b03db]#012 8:
(Paxos::do_refresh()+0
x2e) [0x5eea5e]#012 9: (Paxos::commit_finish()+0x569) [0x5fbf39]#012 10:
(C_Committed::finish(int)+0x2b) [0x60038b]#012 11: (Context::complete(int)+0x9)
[0x5d4d89]#012 12: (MonitorDBStore::C_DoTransaction::finish(int)+0x8c)
[0x5ff4bc]#012 13: (Context::complete(int)+0x9) [0x5d4d89]#012 14:
(Finisher::finisher_thread_entry()+0x158) [0x717e88]#012 15: (()+0x8182)
[0x7f3159b5b182]#012 16: (clone()+0x6d) [0x7f31580c647d]#012 NOTE: a copy of
the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 29 14:29:45 95z3zz1 ceph-mon: --- logging levels ---
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 none
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 lockdep
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 context
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 1 crush
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_balancer
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_locker
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_log
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_log_expire
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mds_migrator
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 buffer
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 timer
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 filer
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 striper
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 1 objecter
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 rados
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 rbd
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 rbd_replay
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 journaler
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 objectcacher
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 client
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 osd
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 optracker
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 objclass
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 3 filestore
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 3 keyvaluestore
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 3 journal
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 ms
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 mon
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/10 monc
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 paxos
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 5 tp
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 auth
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 crypto
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 1 finisher
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 heartbeatmap
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 perfcounter
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 rgw
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/10 civetweb
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 javaclient
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 asok
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 1 throttle
Oct 29 14:29:45 95z3zz1 ceph-mon: 0/ 0 refs
Oct 29 14:29:45 95z3zz1 ceph-mon: 1/ 5 xio
Oct 29 14:29:45 95z3zz1 ceph-mon: 99/99 (syslog threshold)
Oct 29 14:29:45 95z3zz1 ceph-mon: -1/-1 (stderr threshold)
Oct 29 14:29:45 95z3zz1 ceph-mon: max_recent 10000
Oct 29 14:29:45 95z3zz1 ceph-mon: max_new 1000
Oct 29 14:29:45 95z3zz1 ceph-mon: log_file /var/log/ceph/ceph-mon.95z3zz1.log
Oct 29 14:29:45 95z3zz1 ceph-mon: --- end dump of recent events ---
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.806299 7f31ad4e38c0 0
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process
ceph-mon, pid 143457
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819073 7f31ad4e38c0 0
starting mon.95z3zz1 rank 0 at 157.249.160.11:6789/0 mon_data
/var/lib/ceph/mon/ceph-95z3zz1 fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819332 7f31ad4e38c0 1
mon.95z3zz1@-1(probing) e7 preinit fsid 6a57f77c-a465-4fc6-b0a2-93c20371bab8
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819670 7f31ad4e38c0 1
mon.95z3zz1@-1(probing).paxosservice(pgmap 33661126..33661654) refresh
upgraded, format 0 -> 1
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.819682 7f31ad4e38c0 1
mon.95z3zz1@-1(probing).pg v0 on_upgrade discarding in-core PGMap
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.932544 7f31ad4e38c0 0
mon.95z3zz1@-1(probing).mds e89
print_map#012epoch#01189#012flags#0110#012created#0112014-07-31
11:09:25.038824#012modified#0112015-10-28
11:59:17.337799#012tableserver#0110#012root#0110#012session_timeout#01160#012session_autoclose#011300#012max_file_size#0111099511627776#012last_failure#0110#012last_failure_osd_epoch#01161286#012compat#011compat={},rocompat={},incompat={1=base
v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor
table}#012max_mds#0111#012in#0110#012up#011{0=15517917}#012failed#011#012stopped#011#012data_pools#0110#012metadata_pool#0111#012inline_data#011disabled#01215517917:#011157.249.160.107:6800/3998
'56z3zz1' mds.0.13 up:active seq 188
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933110 7f31ad4e38c0 0
mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting
msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933120 7f31ad4e38c0 0
mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting
msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933126 7f31ad4e38c0 0
mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting
msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: 2015-10-29 14:29:45.933129 7f31ad4e38c0 0
mon.95z3zz1@-1(probing).osd e61385 crush map has features 1107558400, adjusting
msgr requires
Oct 29 14:29:45 95z3zz1 ceph-mon: mon.0 157.249.160.11:6789/0 8 : pgmap
v33661654: 21936 pgs: 21936 active+clean; 9824 GB data, 29683 GB used, 144 TB /
173 TB avail; 6259 kB/s rd, 39388 kB/s wr, 4361 op/s
--
Requests: serviced...@met.no
Personal: arnulf.heimsb...@met.no
Phone : +47 22 06 30 00/32 54
Confidential or sensitive, use one of
miniLock: T1zSJJA9dm7M7ECdmvYLe9jxmgGCGAifShcmJ787uDZ6q
PGP ID : AD21343A
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
