Our mons just went into a logging frenzy.

We have 3 mons in the cluster, and they mostly log stuff like this

2015-06-02 18:00:48.749386 7f1c08c0d700  1 mon.node-10@1(peon).paxos(paxos 
active c 36603331..36604063) is_readable now=2015-06-02 18:00:48.749389 
lease_expire=2015-06-02 18:00:53.507837 has v0 lc 36604063
2015-06-02 18:00:49.025179 7f1c08c0d700  1 mon.node-10@1(peon).paxos(paxos 
active c 36603331..36604063) is_readable now=2015-06-02 18:00:49.025187 
lease_expire=2015-06-02 18:00:53.507837 has v0 lc 36604063
2015-06-02 18:00:49.025640 7f1c08c0d700  1 mon.node-10@1(peon).paxos(paxos 
active c 36603331..36604063) is_readable now=2015-06-02 18:00:49.025642 
lease_expire=2015-06-02 18:00:53.507837 has v0 lc 36604063
2015-06-02 18:00:49.026132 7f1c08c0d700  1 mon.node-10@1(peon).paxos(paxos 
active c 36603331..36604063) is_readable now=2015-06-02 18:00:49.026134 
lease_expire=2015-06-02 18:00:53.507837 has v0 lc 36604063
2015-06-02 18:00:49.028388 7f1c08c0d700  1 mon.node-10@1(peon).paxos(paxos 
active c 36603331..36604063) is_readable now=2015-06-02 18:00:49.028393 
lease_expire=2015-06-02 18:00:53.507837 has v0 lc 36604063


There are few lines every second, sometimes more, sometimes less (tell me if 
that’s normal. I’m not sure)

Two of them went completely haywire, one log is 17GB now and rising. It’s still 
mostly the same content, just more frequent:

2015-06-02 18:09:00.879950 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
updating c 36604084..36604772) is_readable now=2015-06-02 18:09:00.879956 
lease_expire=0.000000 has v0 lc 36604772
2015-06-02 18:09:00.879968 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
updating c 36604084..36604772) is_readable now=2015-06-02 18:09:00.879969 
lease_expire=0.000000 has v0 lc 36604772
2015-06-02 18:09:00.954835 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
updating c 36604084..36604772) is_readable now=2015-06-02 18:09:00.954843 
lease_expire=0.000000 has v0 lc 36604772
2015-06-02 18:09:00.954860 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
updating c 36604084..36604772) is_readable now=2015-06-02 18:09:00.954861 
lease_expire=0.000000 has v0 lc 36604772
2015-06-02 18:09:01.249648 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
active c 36604084..36604773) is_readable now=2015-06-02 18:09:01.249668 
lease_expire=2015-06-02 18:09:06.091738 has v0 lc 36604773
2015-06-02 18:09:01.249697 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
active c 36604084..36604773) is_readable now=2015-06-02 18:09:01.249699 
lease_expire=2015-06-02 18:09:06.091738 has v0 lc 36604773
2015-06-02 18:09:01.249708 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
active c 36604084..36604773) is_readable now=2015-06-02 18:09:01.249709 
lease_expire=2015-06-02 18:09:06.091738 has v0 lc 36604773
2015-06-02 18:09:01.249736 7f4309d45700  1 mon.node-14@2(peon).paxos(paxos 
active c 36604084..36604773) is_readable now=2015-06-02 18:09:01.249736 
lease_expire=2015-06-02 18:09:06.091738 has v0 lc 36604773


Any idea what it might be? Clocks look synced, no other aparent problem that I 
can see, the cluster is working.
I’d like to know why this happened before I restart the unhealthy mons which (I 
hope) will fix this.

Thanks

Jan

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to