[ceph-users] 答复: 答复: calculate past_intervals wrong, lead to choose wrong authority osd, then osd assert(newhead >= log.tail)

[Chenyehua]

Reproduce as Follows:
 HOST-A   HOST-B  HOST-C
 osd 7  osd 21   osd11
1. shutdown HOST-C
2. for a long time, cluster has only HOST A and HOST B, write data 3. shutdown 
HOST-A => then start HOST-C=> restart ceph-osd-all on HOST-B about 5 times,at 
the same time start HOST-A,restart ceph-osd-all on HOST-A serveral times

Chage some config :
[mon]
mon_min_osdmap_epochs = 10
paxos_service_trim_min = 5
paxos_service_trim_max = 10
[osd]
osd_min_pg_log_entries = 1
osd_max_pg_log_entries = 1
osd_pg_log_trim_min = 1


this time , the problem can be reproduced much more easily （hammer ）, and I 
pushed PR# 16371, please have a look, thanks.

Best regards

-邮件原件-
发件人: Sage Weil [mailto:sw...@redhat.com] 
发送时间: 2017年7月14日 21:49
收件人: chenyehua 11692 (RD)
抄送: 'ceph-us...@ceph.com'
主题: Re: 答复: calculate past_intervals wrong, lead to choose wrong authority osd, 
then osd assert(newhead >= log.tail) 

On Fri, 14 Jul 2017, Chenyehua wrote:
> Thanks, Sage.
> 
> It doesn't happen every time, but the probability is high
> 
> Reproduce as Follows:
> HOST-A   HOST-B  HOST-C
> osd 7  osd 21   osd11
>  1. osdmap epoch95, pg 1.20f on osd acting set [11,7]/ up 
> set[11,7],then shutdown HOST-C  2. for a long time, cluster has only 
> HOST A and HOST B, write data  3. shutdown HOST-A , then start HOST-C, 
> restart HOST-B about 4 times  4. start HOST-A, osd 7 assert  And set 
> pg log configuration:
> "osd_min_pg_log_entries": "100",
> "osd_max_pg_log_entries": "300",
> "osd_pg_log_trim_min": "100"
> 
> After analyzing ceph-osd.log, I think the root cause is "some osd compute the 
> wrong past interval"
> 
> In my test ,osd 11 has down for a long time ,it has very old data, when it 
> came up, first received full_1001 osdmap,and generate past_interval as 
> follows:
> 92~1000 [11,7]/[11,7]this is wrong interval; Actually 
> during osdmap 92 to 1000, pg already become active+clean on [7,21],and data 
> has updated a lot; 
> this past interval is inappropriate and could lead to osd11 think "I've been 
> alive and is the primary during osdmap 92~1000,  already have the same data 
> epoch with osd 7"
> next osdmap epoch, pg mapped only on [11], this pg can become active and 
> modify the last_epoch_start; In the future, other osd who has newest data 
> come back, find_best_info would choose out the one who has bigger 
> last_epoch_start, yeah it's osd 11; finally osd who has the older data 
> unexpectly become the best.

Ooh... I think this is because the past intervals generation can't really deal 
with the osdmap gap.  You can make it easier to trigger this case by setting 
mon_min_osdmap_epochs = 50 or some other smaller number (the default is 500).  
(You can, conversely, make this very hard to trigger by setting it to a value 
larger than 500, at the expense of some mon disk
space.)

Can you try reproducing with a lower value?  If we can reliably reproduce this 
I'm confident we can come up with a real fix (for hammer and jewel, if it is 
also affected).  I'm pretty sure luminous is not since the past intervals code 
was almost completely restructured, but with a method to reproduce we can 
confirm.

Thank you!
sage


> 
> Besides, after using the ceph-objectstore-tool import/export, the cluster 
> became healthy and all pgs were active+clean, however, the client io still 
> failed.
> 
> I have not tested on jewel, but looked at Jewel code, I think this 
> problem seems still exists; later I will test it;
> 
> I want to solve this problem in hammer branch, what should I do? Could you 
> give me some advice? Thanks.
> 
> 
> 
> -邮件原件-
> 发件人: Sage Weil [mailto:sw...@redhat.com]
> 发送时间: 2017年7月13日 22:51
> 收件人: chenyehua 11692 (RD)
> 抄送: 'ceph-us...@ceph.com'
> 主题: Re: calculate past_intervals wrong, lead to choose wrong authority 
> osd, then osd assert(newhead >= log.tail)
> 
> Hi Chenyehua,
> 
> This looks a lot like a problem we've seen several times on hammer and never 
> managed to find a root cause for.  Is this something that happened once or 
> can you reproduce it?  To my knowledge it has not happened on jewel, so my 
> first suggestion is to upgrade (hammer is pretty old now :).
> 
> Are you looking for help on resolving this specific issue for this cluster?  
> In the past we've used a combination of ceph-objectstore-tool import/export 
> and/or osd_find_best_info_ignore_history_les on the primary to resolve it.
> 
> sage
> 
> 
> 
> On Thu, 13 Jul 2017, Chenyehua wrote:
> 
> > 
> > Hi Sage
> > 
> > I find  the osd as

[ceph-users] 答复: calculate past_intervals wrong, lead to choose wrong authority osd, then osd assert(newhead >= log.tail)

[Chenyehua]

Thanks, Sage.

It doesn't happen every time, but the probability is high 

Reproduce as Follows:
HOST-A   HOST-B  HOST-C
osd 7  osd 21   osd11
 1. osdmap epoch95, pg 1.20f on osd acting set [11,7]/ up set[11,7],then 
shutdown HOST-C 
 2. for a long time, cluster has only HOST A and HOST B, write data 
 3. shutdown HOST-A , then start HOST-C, restart HOST-B about 4 times 
 4. start HOST-A, osd 7 assert
 And set pg log configuration:
"osd_min_pg_log_entries": "100",
"osd_max_pg_log_entries": "300",
"osd_pg_log_trim_min": "100"

After analyzing ceph-osd.log, I think the root cause is "some osd compute the 
wrong past interval"

In my test ,osd 11 has down for a long time ,it has very old data, when it came 
up, first received full_1001 osdmap,and generate past_interval as follows:
92~1000 [11,7]/[11,7]this is wrong interval; Actually 
during osdmap 92 to 1000, pg already become active+clean on [7,21],and data has 
updated a lot; 
this past interval is inappropriate and could lead to osd11 think "I've been 
alive and is the primary during osdmap 92~1000,  already have the same data 
epoch with osd 7"
next osdmap epoch, pg mapped only on [11], this pg can become active and modify 
the last_epoch_start; In the future, other osd who has newest data come back, 
find_best_info would choose out the one who has bigger last_epoch_start, yeah 
it's osd 11; finally osd who has the older data unexpectly become the best.

Besides, after using the ceph-objectstore-tool import/export, the cluster 
became healthy and all pgs were active+clean, however, the client io still 
failed.

I have not tested on jewel, but looked at Jewel code, I think this problem 
seems still exists; later I will test it;

I want to solve this problem in hammer branch, what should I do? Could you give 
me some advice? Thanks.



-邮件原件-
发件人: Sage Weil [mailto:sw...@redhat.com] 
发送时间: 2017年7月13日 22:51
收件人: chenyehua 11692 (RD)
抄送: 'ceph-us...@ceph.com'
主题: Re: calculate past_intervals wrong, lead to choose wrong authority osd, 
then osd assert(newhead >= log.tail) 

Hi Chenyehua,

This looks a lot like a problem we've seen several times on hammer and never 
managed to find a root cause for.  Is this something that happened once or can 
you reproduce it?  To my knowledge it has not happened on jewel, so my first 
suggestion is to upgrade (hammer is pretty old now :).

Are you looking for help on resolving this specific issue for this cluster?  In 
the past we've used a combination of ceph-objectstore-tool import/export and/or 
osd_find_best_info_ignore_history_les on the primary to resolve it.

sage



On Thu, 13 Jul 2017, Chenyehua wrote:

> 
> Hi Sage
> 
> I find  the osd assert due to the  wrongly  generated past_intervals, 
> could you give me some advice and solutions to this problem?
> 
>  
> 
> Here is the detail:
> 
>  
> 
> Ceph version: 0.94.5
> 
>  
> 
> HOST-A   HOST-B    HOST-C
> 
> osd 7    osd 21   osd11
> 
> 1. osdmap epoch95, pg 1.20f on osd acting set [11,7]/ up 
> set[11,7],then shutdown HOST-C
> 
> 2. for a long time, cluster has only HOST A and HOST B, write data
> 
> 3. shutdown HOST-A , then start HOST-C, restart HOST-B about 4 times
> 
> 4. start HOST-A, osd 21 assert
> 
>  
> 
> Analysis:
> 
> when osd 11 start, it generate past_intervals wrongly, make [92~1000] 
> in the same interval
> 
> pg map 1673,osd11 become the primary,and pg 1.20f change from peering 
> to
> activating+undersized+degraded , modified last_epoch_start;
> 
> osd7 start, find_best_info will choose out bigger
> last_epoch_start，althought osd7 has the latest data;
> 
> past_intervals on osd 7:
> 
> ~95     [11,7]/[11,7]
> 
> 96~100    [7]/[7]
> 
> 101     [7,21]/[7,21]
> 
> 102~178     [7,21]/[7]
> 
> 179~1663  [7,21]/[7,21]
> 
> 1664~1672  [21]/[21]
> 
> 1673~1692  [11]/[11]
> 
>  
> 
> past_intervals on osd11:
> 
> 92~1000     [11,7]/[11,7]the wrong pi
> 
> 1001~1663   [7,21]/[7,21] no rw
> 
> 1664~1672   [21]/[21] no rw
> 
> 1673~1692    [11]/[11]
> 
>  
> 
>  
> 
>  
> 
> Logs:
> 
> Assert on osd7:
> 
> 2017-07-10 16:08:29.836722 7f4fac24a700 -1 osd/PGLog.cc: In function 
> 'void PGLog::rewind_divergent_log(ObjectStore::Transaction&, 
> eversion_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)' thread 
> 7f4fac24a700 time
> 2017-07-10 16:08:29.833699
> 
> osd/PGLog.cc: 503: FAILED assert(newhead >= log.tail)
> 
> ceph version 0.94.5 (664cc0b54fdb496233a81ab19d42df3f46dcda50)
&g

[ceph-users] calculate past_intervals wrong, lead to choose wrong authority osd, then osd assert(newhead >= log.tail)

[Chenyehua]

Hi Sage
I find  the osd assert due to the  wrongly  generated past_intervals, could you 
give me some advice and solutions to this problem?

Here is the detail:

Ceph version: 0.94.5

HOST-A   HOST-BHOST-C
osd 7osd 21   osd11
1. osdmap epoch95, pg 1.20f on osd acting set [11,7]/ up set[11,7],then 
shutdown HOST-C
2. for a long time, cluster has only HOST A and HOST B, write data
3. shutdown HOST-A , then start HOST-C, restart HOST-B about 4 times
4. start HOST-A, osd 21 assert

Analysis:
when osd 11 start, it generate past_intervals wrongly, make [92~1000] in the 
same interval
pg map 1673,osd11 become the primary,and pg 1.20f change from peering to 
activating+undersized+degraded , modified last_epoch_start;
osd7 start, find_best_info will choose out bigger last_epoch_start，althought 
osd7 has the latest data;
past_intervals on osd 7:
~95 [11,7]/[11,7]
96~100[7]/[7]
101 [7,21]/[7,21]
102~178 [7,21]/[7]
179~1663  [7,21]/[7,21]
1664~1672  [21]/[21]
1673~1692  [11]/[11]

past_intervals on osd11:
92~1000 [11,7]/[11,7]the wrong pi
1001~1663   [7,21]/[7,21] no rw
1664~1672   [21]/[21] no rw
1673~1692[11]/[11]



Logs:
Assert on osd7:
2017-07-10 16:08:29.836722 7f4fac24a700 -1 osd/PGLog.cc: In function 'void 
PGLog::rewind_divergent_log(ObjectStore::Transaction&, eversion_t, pg_info_t&, 
PGLog::LogEntryHandler*, bool&, bool&)' thread 7f4fac24a700 time 2017-07-10 
16:08:29.833699
osd/PGLog.cc: 503: FAILED assert(newhead >= log.tail)
ceph version 0.94.5 (664cc0b54fdb496233a81ab19d42df3f46dcda50)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) 
[0xbd1ebb]
2: (PGLog::rewind_divergent_log(ObjectStore::Transaction&, eversion_t, 
pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0x60b) [0x7840fb]
3: (PG::rewind_divergent_log(ObjectStore::Transaction&, eversion_t)+0x97) 
[0x7df4b7]
4: (PG::RecoveryState::Stray::react(PG::MInfoRec const&)+0x22f) [0x80109f]
5: (boost::statechart::simple_state, 
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base 
const&, void const*)+0x216) [0x83c8d6]
6: (boost::statechart::state_machine, 
boost::statechart::null_exception_translator>::send_event(boost::statechart::event_base
 const&)+0x5b) [0x827edb]
7: (PG::handle_peering_event(std::tr1::shared_ptr, 
PG::RecoveryCtx*)+0x1ce) [0x7d5dce]
8: (OSD::process_peering_events(std::list > const&, 
ThreadPool::TPHandle&)+0x2c0) [0x6b5930]
9: (OSD::PeeringWQ::_process(std::list > const&, 
ThreadPool::TPHandle&)+0x18) [0x70ef18]
10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xbc2aa6]
11: (ThreadPool::WorkThread::entry()+0x10) [0xbc3b50]
12: (()+0x8182) [0x7f4fc6458182]
13: (clone()+0x6d) [0x7f4fc49c347d]


osd 11 log:
2017-07-10 15:40:43.742547 7f8b1567a740 10 osd.11 95 pgid 1.20f coll 1.20f_head
2017-07-10 15:40:43.742580 7f8b1567a740 10 osd.11 95 _open_lock_pg 1.20f
2017-07-10 15:40:43.742592 7f8b1567a740  5 osd.11 pg_epoch: 95 
pg[1.20f(unlocked)] enter Initial
2017-07-10 15:40:43.743207 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=0 crt=95'394 lcod 0'0 mlcod 0'0 inactive] handle_loaded
2017-07-10 15:40:43.743214 7f8b1567a740  5 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=0 crt=95'394 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.000622 0 0.00
2017-07-10 15:40:43.743220 7f8b1567a740  5 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=0 crt=95'394 lcod 0'0 mlcod 0'0 inactive] enter Reset
2017-07-10 15:40:43.743224 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] Clearing blocked outgoing 
recovery messages
2017-07-10 15:40:43.743228 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] Not blocking outgoing recovery 
messages
2017-07-10 15:40:43.743232 7f8b1567a740 10 osd.11 95 load_pgs loaded pg[1.20f( 
v 95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] 
r=0 lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] log((95'300,95'397], 
crt=95'394)
2017-07-10 15:40:43.829867 7f8b1567a740 10 osd.11 pg_epoch: 95 
pg[1.20f(unlocked)] _calc_past_interval_range start epoch 93 >= end epoch 92, 
nothing to do
2017-07-10 15:42:40.899157 7f8b1567a740 10 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=95 crt=95'394 lcod 0'0 mlcod 0'0 inactive] null
2017-07-10 15:42:40.902520 7f8afa1bd700 10 osd.11 pg_epoch: 95 pg[1.20f( v 
95'397 (95'300,95'397] local-les=93 n=69 ec=75 les/c 93/93 92/92/92) [11,7] r=0 
lpr=95 crt=9

[ceph-users] 3mon cluster, after ifdown&ifup the public network interface of leader mon, sendQ of one peon monitor will suddenly increase sharply

[Chenyehua]

Dear Cephers:
I meet a problem now:
my cluster has 3 servers, each server has 1mon and 1osd
node0(mon0+osd0)、node1（mon1+osd1）、node2（mon2+osd2）

3 mon(mon0 192.168.202.35/24, mon1 192.168.202.36/24, mon2 192.168.202.37/24)

public network 192.168.2.*/24
cluster network 172.16.2.*/24

operation:
1.ifdown node0 public network interface for 30s
2.on node1, netstat -apn-c 1 |grep 6789, you can see the sendQ of tcp 
connection (src: mon1 -> dst: mon0) would be up to several hundred or several 
thousand ;
3.ifup node0's public network interface, use netstat again on node1 
,suddenly the sendQ can be up to 100 thousand or more; this will lead to in the 
following several seconds, mon1 can not send defer message to mon0(because of 
tcp sendQ block), this time "ceph -s" will display mon1 is down;
until sendQ of mon1 back to zero, 3 mon can finish election;

my questions:
1.why tcp sendQ(mon1 prepare to send to mon0) suddenly increase sharply when 
ifup mon0's public network?
2.when ifdown node0 net interface, state of the tcp conn (between mon0 and 
mon1) still be ESTABLISHED, after 900s ,conn close;
but if I down cluster network interface of node0, tcp conn (between osd0 and 
osd1) almost turn to be TIME_WAIT1;
why mon-mon tcp conn policy is different from osd-osd tcp conn?

Looking forward to your reply，thanks！
Best regards！

-
本邮件及其附件含有新华三技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from New H3C, 
which is
intended only for the person or entity whose address is listed above. Any use 
of the
information contained herein in any way (including, but not limited to, total 
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender
by phone or email immediately and delete it!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 答复: leveldb takes a lot of space

[Chenyehua]

@ Niv Azriel  : What is your leveldb version and has it been fixed now?
@ Wido den Hollander : I also meet a similar problem:
the size of my leveldb is about 
17GB(300+ osds), there are a lot of sst files(each sst file is 2MB) in 
/var/lib/ceph/mon. (a network abnormal situation once happened)
The leveldb version is 1.2 (ubuntu 
12.04, ceph 0.94.5)

-邮件原件-
发件人: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] 代表 Wido den Hollander
发送时间: 2017年3月27日 19:30
收件人: ceph-users@lists.ceph.com; Niv Azriel
主题: Re: [ceph-users] leveldb takes a lot of space


> Op 26 maart 2017 om 9:44 schreef Niv Azriel :
>
>
> after network issues, ceph cluster fails.
> leveldb grows and takes a lot of space ceph mon cant write to leveldb
> because there is not enough space on filesystem.
> (there is a lot of ldb file on /var/lib/ceph/mon)
>

It is normal that the database will grow as the MON will keep all historic 
OSDMaps when one or more PGs are not active+clean

> ceph compact on start is not helping.
> my erasure-code is too big.
>
> how to fix it?

Make sure you have enough space available on your MONs, that is the main 
advise. Under normal operations <2GB should be enough, but it can grow much 
bigger.

On most clusters I design I make sure there is at least 200GB of space 
available on each MON on a fast DC-grade SSD.

Wido

> thanks in advanced
>
> ceph version: jewel
> os : ubuntu16.04
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-
本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from H3C, 
which is
intended only for the person or entity whose address is listed above. Any use 
of the
information contained herein in any way (including, but not limited to, total 
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender
by phone or email immediately and delete it!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 答复: 答复: mon is stuck in leveldb and costs nearly 100% cpu

[Chenyehua]

Thanks for the response, Shinobu
The warning disappears due to your suggesting solution, however the nearly 100% 
cpu cost still exists and concerns me a lot.
So, do you know why the cpu cost is so high?
Are there any solutions or suggestions to this problem?

Cheers

-邮件原件-
发件人: Shinobu Kinjo [mailto:ski...@redhat.com] 
发送时间: 2017年2月13日 10:54
收件人: chenyehua 11692 (RD)
抄送: kc...@redhat.com; ceph-users@lists.ceph.com
主题: Re: 答复: [ceph-users] mon is stuck in leveldb and costs nearly 100% cpu

O.k, that's reasonable answer. Would you do on all hosts which the MON are 
running on:

 #* ceph --admin-daemon /var/run/ceph/ceph-mon.`hostname -s`.asok config show | 
grep leveldb_log

Anyway you can compact leveldb size with at runtime:

 #* ceph tell mon.`hostname -s` compact

And you should set in ceph.conf to prevent same issue from the next:

 #* [mon]
 #* mon compact on start = true


On Mon, Feb 13, 2017 at 11:37 AM, Chenyehua  wrote:
> Sorry, I made a mistake, the ceph version is actually 0.94.5
>
> -邮件原件-
> 发件人: chenyehua 11692 (RD)
> 发送时间: 2017年2月13日 9:40
> 收件人: 'Shinobu Kinjo'
> 抄送: kc...@redhat.com; ceph-users@lists.ceph.com
> 主题: 答复: [ceph-users] mon is stuck in leveldb and costs nearly 100% cpu
>
> My ceph version is 10.2.5
>
> -邮件原件-
> 发件人: Shinobu Kinjo [mailto:ski...@redhat.com]
> 发送时间: 2017年2月12日 13:12
> 收件人: chenyehua 11692 (RD)
> 抄送: kc...@redhat.com; ceph-users@lists.ceph.com
> 主题: Re: [ceph-users] mon is stuck in leveldb and costs nearly 100% cpu
>
> Which Ceph version are you using?
>
> On Sat, Feb 11, 2017 at 5:02 PM, Chenyehua  wrote:
>> Dear Mr Kefu Chai
>>
>> Sorry to disturb you.
>>
>> I meet a problem recently. In my ceph cluster ,health status has 
>> warning “store is getting too big!” for several days; and  ceph-mon 
>> costs nearly 100% cpu;
>>
>> Have you ever met this situation?
>>
>> Some detailed information are attached below:
>>
>>
>>
>> root@cvknode17:~# ceph -s
>>
>> cluster 04afba60-3a77-496c-b616-2ecb5e47e141
>>
>>  health HEALTH_WARN
>>
>> mon.cvknode17 store is getting too big! 34104 MB >= 15360 
>> MB
>>
>>  monmap e1: 3 mons at
>> {cvknode15=172.16.51.15:6789/0,cvknode16=172.16.51.16:6789/0,cvknode1
>> 7
>> =172.16.51.17:6789/0}
>>
>> election epoch 862, quorum 0,1,2
>> cvknode15,cvknode16,cvknode17
>>
>>  osdmap e196279: 347 osds: 347 up, 347 in
>>
>>   pgmap v5891025: 33272 pgs, 16 pools, 26944 GB data, 6822 
>> kobjects
>>
>> 65966 GB used, 579 TB / 644 TB avail
>>
>>33270 active+clean
>>
>>2 active+clean+scrubbing+deep
>>
>>   client io 840 kB/s rd, 739 kB/s wr, 35 op/s rd, 184 op/s wr
>>
>>
>>
>> root@cvknode17:~# top
>>
>> top - 15:19:28 up 23 days, 23:58,  6 users,  load average: 1.08, 
>> 1.40,
>> 1.77
>>
>> Tasks: 346 total,   2 running, 342 sleeping,   0 stopped,   2 zombie
>>
>> Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  
>> 2.5%si, 0.0%st
>>
>> Mem:  65384424k total, 58102880k used,  7281544k free,   240720k buffers
>>
>> Swap: 2100k total,   344944k used, 29654156k free, 24274272k cached
>>
>>
>>
>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>>
>>   24407 root  20   0 17.3g  12g  10m S   98 20.2   8420:11 ceph-mon
>>
>>
>>
>> root@cvknode17:~# top -Hp 24407
>>
>> top - 15:19:49 up 23 days, 23:59,  6 users,  load average: 1.12, 
>> 1.39,
>> 1.76
>>
>> Tasks:  17 total,   1 running,  16 sleeping,   0 stopped,   0 zombie
>>
>> Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  
>> 2.5%si, 0.0%st
>>
>> Mem:  65384424k total, 58104868k used,  7279556k free,   240744k buffers
>>
>> Swap: 2100k total,   344944k used, 29654156k free, 24271188k cached
>>
>>
>>
>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>>
>>   25931 root  20   0 17.3g  12g   9m R   98 20.2   7957:37 ceph-mon
>>
>>   24514 root  20   0 17.3g  12g   9m S2 20.2   3:06.75 ceph-mon
>>
>>   25932 root  20   0 17.3g  12g   9m S2 20.2   1:07.82 ceph-mon
>>
>>   24407 root  20   0 17.3g  12g   9m S0 20.2   0:00.67 ceph-mon
>>
>>   24508 root  20   0 17.3g  12g   9m S0 20.2  15:50.24 ceph-mon
>>
>>   24513 root  20   0 17.3g  12g   9m S0 20.2   0:07.88 ceph-mon
>>
>>   24534 

[ceph-users] 答复: mon is stuck in leveldb and costs nearly 100% cpu

[Chenyehua]

Sorry, I made a mistake, the ceph version is actually 0.94.5

-邮件原件-
发件人: chenyehua 11692 (RD) 
发送时间: 2017年2月13日 9:40
收件人: 'Shinobu Kinjo'
抄送: kc...@redhat.com; ceph-users@lists.ceph.com
主题: 答复: [ceph-users] mon is stuck in leveldb and costs nearly 100% cpu

My ceph version is 10.2.5

-邮件原件-
发件人: Shinobu Kinjo [mailto:ski...@redhat.com]
发送时间: 2017年2月12日 13:12
收件人: chenyehua 11692 (RD)
抄送: kc...@redhat.com; ceph-users@lists.ceph.com
主题: Re: [ceph-users] mon is stuck in leveldb and costs nearly 100% cpu

Which Ceph version are you using?

On Sat, Feb 11, 2017 at 5:02 PM, Chenyehua  wrote:
> Dear Mr Kefu Chai
>
> Sorry to disturb you.
>
> I meet a problem recently. In my ceph cluster ,health status has 
> warning “store is getting too big!” for several days; and  ceph-mon 
> costs nearly 100% cpu;
>
> Have you ever met this situation?
>
> Some detailed information are attached below:
>
>
>
> root@cvknode17:~# ceph -s
>
> cluster 04afba60-3a77-496c-b616-2ecb5e47e141
>
>  health HEALTH_WARN
>
> mon.cvknode17 store is getting too big! 34104 MB >= 15360 
> MB
>
>  monmap e1: 3 mons at
> {cvknode15=172.16.51.15:6789/0,cvknode16=172.16.51.16:6789/0,cvknode17
> =172.16.51.17:6789/0}
>
> election epoch 862, quorum 0,1,2
> cvknode15,cvknode16,cvknode17
>
>  osdmap e196279: 347 osds: 347 up, 347 in
>
>   pgmap v5891025: 33272 pgs, 16 pools, 26944 GB data, 6822 
> kobjects
>
> 65966 GB used, 579 TB / 644 TB avail
>
>33270 active+clean
>
>2 active+clean+scrubbing+deep
>
>   client io 840 kB/s rd, 739 kB/s wr, 35 op/s rd, 184 op/s wr
>
>
>
> root@cvknode17:~# top
>
> top - 15:19:28 up 23 days, 23:58,  6 users,  load average: 1.08, 1.40,
> 1.77
>
> Tasks: 346 total,   2 running, 342 sleeping,   0 stopped,   2 zombie
>
> Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  2.5%si, 
> 0.0%st
>
> Mem:  65384424k total, 58102880k used,  7281544k free,   240720k buffers
>
> Swap: 2100k total,   344944k used, 29654156k free, 24274272k cached
>
>
>
> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>
>   24407 root  20   0 17.3g  12g  10m S   98 20.2   8420:11 ceph-mon
>
>
>
> root@cvknode17:~# top -Hp 24407
>
> top - 15:19:49 up 23 days, 23:59,  6 users,  load average: 1.12, 1.39,
> 1.76
>
> Tasks:  17 total,   1 running,  16 sleeping,   0 stopped,   0 zombie
>
> Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  2.5%si, 
> 0.0%st
>
> Mem:  65384424k total, 58104868k used,  7279556k free,   240744k buffers
>
> Swap: 2100k total,   344944k used, 29654156k free, 24271188k cached
>
>
>
> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>
>   25931 root  20   0 17.3g  12g   9m R   98 20.2   7957:37 ceph-mon
>
>   24514 root  20   0 17.3g  12g   9m S2 20.2   3:06.75 ceph-mon
>
>   25932 root  20   0 17.3g  12g   9m S2 20.2   1:07.82 ceph-mon
>
>   24407 root  20   0 17.3g  12g   9m S0 20.2   0:00.67 ceph-mon
>
>   24508 root  20   0 17.3g  12g   9m S0 20.2  15:50.24 ceph-mon
>
>   24513 root  20   0 17.3g  12g   9m S0 20.2   0:07.88 ceph-mon
>
>   24534 root  20   0 17.3g  12g   9m S0 20.2 196:33.85 ceph-mon
>
>   24535 root  20   0 17.3g  12g   9m S0 20.2   0:00.01 ceph-mon
>
>   25929 root  20   0 17.3g  12g   9m S0 20.2   3:06.09 ceph-mon
>
>   25930 root  20   0 17.3g  12g   9m S0 20.2   8:12.58 ceph-mon
>
>   25933 root  20   0 17.3g  12g   9m S0 20.2   4:42.22 ceph-mon
>
>   25934 root  20   0 17.3g  12g   9m S0 20.2  40:53.27 ceph-mon
>
>   25935 root  20   0 17.3g  12g   9m S0 20.2   0:04.84 ceph-mon
>
>   25936 root  20   0 17.3g  12g   9m S0 20.2   0:00.01 ceph-mon
>
>   25980 root  20   0 17.3g  12g   9m S0 20.2   0:06.65 ceph-mon
>
>   25986 root  20   0 17.3g  12g   9m S0 20.2  48:26.77 ceph-mon
>
>   55738 root  20   0 17.3g  12g   9m S0 20.2   0:09.06 ceph-mon
>
>
>
>
>
> Thread 20 (Thread 0x7f3e77e80700 (LWP 25931)):
>
> #0  0x7f3e7e83a653 in pread64 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
>
> #1  0x009286cf in ?? ()
>
> #2  0x0092c187 in
> leveldb::ReadBlock(leveldb::RandomAccessFile*,
> leveldb::ReadOptions const&, leveldb::BlockHandle const&,
> leveldb::Block**)
> ()
>
> #3  0x00922f41 in leveldb::Table::BlockReader(void*,
> leveldb::ReadOptions const&, leveldb::Slice const&) ()
>
> #4  0x00924840 in ?? ()
>
> #5

[ceph-users] 答复: mon is stuck in leveldb and costs nearly 100% cpu

[Chenyehua]

My ceph version is 10.2.5

-邮件原件-
发件人: Shinobu Kinjo [mailto:ski...@redhat.com] 
发送时间: 2017年2月12日 13:12
收件人: chenyehua 11692 (RD)
抄送: kc...@redhat.com; ceph-users@lists.ceph.com
主题: Re: [ceph-users] mon is stuck in leveldb and costs nearly 100% cpu

Which Ceph version are you using?

On Sat, Feb 11, 2017 at 5:02 PM, Chenyehua  wrote:
> Dear Mr Kefu Chai
>
> Sorry to disturb you.
>
> I meet a problem recently. In my ceph cluster ,health status has 
> warning “store is getting too big!” for several days; and  ceph-mon 
> costs nearly 100% cpu;
>
> Have you ever met this situation?
>
> Some detailed information are attached below:
>
>
>
> root@cvknode17:~# ceph -s
>
> cluster 04afba60-3a77-496c-b616-2ecb5e47e141
>
>  health HEALTH_WARN
>
> mon.cvknode17 store is getting too big! 34104 MB >= 15360 
> MB
>
>  monmap e1: 3 mons at
> {cvknode15=172.16.51.15:6789/0,cvknode16=172.16.51.16:6789/0,cvknode17
> =172.16.51.17:6789/0}
>
> election epoch 862, quorum 0,1,2 
> cvknode15,cvknode16,cvknode17
>
>  osdmap e196279: 347 osds: 347 up, 347 in
>
>   pgmap v5891025: 33272 pgs, 16 pools, 26944 GB data, 6822 
> kobjects
>
> 65966 GB used, 579 TB / 644 TB avail
>
>33270 active+clean
>
>2 active+clean+scrubbing+deep
>
>   client io 840 kB/s rd, 739 kB/s wr, 35 op/s rd, 184 op/s wr
>
>
>
> root@cvknode17:~# top
>
> top - 15:19:28 up 23 days, 23:58,  6 users,  load average: 1.08, 1.40, 
> 1.77
>
> Tasks: 346 total,   2 running, 342 sleeping,   0 stopped,   2 zombie
>
> Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  2.5%si, 
> 0.0%st
>
> Mem:  65384424k total, 58102880k used,  7281544k free,   240720k buffers
>
> Swap: 2100k total,   344944k used, 29654156k free, 24274272k cached
>
>
>
> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>
>   24407 root  20   0 17.3g  12g  10m S   98 20.2   8420:11 ceph-mon
>
>
>
> root@cvknode17:~# top -Hp 24407
>
> top - 15:19:49 up 23 days, 23:59,  6 users,  load average: 1.12, 1.39, 
> 1.76
>
> Tasks:  17 total,   1 running,  16 sleeping,   0 stopped,   0 zombie
>
> Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  2.5%si, 
> 0.0%st
>
> Mem:  65384424k total, 58104868k used,  7279556k free,   240744k buffers
>
> Swap: 2100k total,   344944k used, 29654156k free, 24271188k cached
>
>
>
> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
>
>   25931 root  20   0 17.3g  12g   9m R   98 20.2   7957:37 ceph-mon
>
>   24514 root  20   0 17.3g  12g   9m S2 20.2   3:06.75 ceph-mon
>
>   25932 root  20   0 17.3g  12g   9m S2 20.2   1:07.82 ceph-mon
>
>   24407 root  20   0 17.3g  12g   9m S0 20.2   0:00.67 ceph-mon
>
>   24508 root  20   0 17.3g  12g   9m S0 20.2  15:50.24 ceph-mon
>
>   24513 root  20   0 17.3g  12g   9m S0 20.2   0:07.88 ceph-mon
>
>   24534 root  20   0 17.3g  12g   9m S0 20.2 196:33.85 ceph-mon
>
>   24535 root  20   0 17.3g  12g   9m S0 20.2   0:00.01 ceph-mon
>
>   25929 root  20   0 17.3g  12g   9m S0 20.2   3:06.09 ceph-mon
>
>   25930 root  20   0 17.3g  12g   9m S0 20.2   8:12.58 ceph-mon
>
>   25933 root  20   0 17.3g  12g   9m S0 20.2   4:42.22 ceph-mon
>
>   25934 root  20   0 17.3g  12g   9m S0 20.2  40:53.27 ceph-mon
>
>   25935 root  20   0 17.3g  12g   9m S0 20.2   0:04.84 ceph-mon
>
>   25936 root  20   0 17.3g  12g   9m S0 20.2   0:00.01 ceph-mon
>
>   25980 root  20   0 17.3g  12g   9m S0 20.2   0:06.65 ceph-mon
>
>   25986 root  20   0 17.3g  12g   9m S0 20.2  48:26.77 ceph-mon
>
>   55738 root  20   0 17.3g  12g   9m S0 20.2   0:09.06 ceph-mon
>
>
>
>
>
> Thread 20 (Thread 0x7f3e77e80700 (LWP 25931)):
>
> #0  0x7f3e7e83a653 in pread64 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
>
> #1  0x009286cf in ?? ()
>
> #2  0x0092c187 in 
> leveldb::ReadBlock(leveldb::RandomAccessFile*,
> leveldb::ReadOptions const&, leveldb::BlockHandle const&, 
> leveldb::Block**)
> ()
>
> #3  0x00922f41 in leveldb::Table::BlockReader(void*,
> leveldb::ReadOptions const&, leveldb::Slice const&) ()
>
> #4  0x00924840 in ?? ()
>
> #5  0x00924b39 in ?? ()
>
> #6  0x00924a7a in ?? ()
>
> #7  0x009227d0 in ?? ()
>
> #8  0x009140b6 in ?? ()
>
> #9  0x009143dd in ?? ()
>
> #10 0x0088d399 in
> LevelDBStore::LevelDBWholeSpace

[ceph-users] libcephfs prints error" auth method 'x' error -1 "

[Chenyehua]

Dear Mr Kefu Chai
Sorry to disturb you again, I meet another problem:

Mon has several sessions, one of which seems to be something wrong. The 
authentication fails and libcephfs keeps printing “libceph: auth method 'x' 
error -1”

It may be relevant to the unexpected key, but I do not know the exact reasons

Have your ever met this situation or known the exact reasons?

The abnormal logs are attached:



Libcephfs print error:

Sat Jan 21 14:17:12 2017] libceph: auth method 'x' error -1
[Sat Jan 21 14:17:22 2017] libceph: auth method 'x' error -1
[Sat Jan 21 14:17:32 2017] libceph: auth method 'x' error -1
[Sat Jan 21 14:17:42 2017] libceph: auth method 'x' error -1
[Sat Jan 21 14:17:52 2017] libceph: auth method 'x' error -1
[Sat Jan 21 14:18:02 2017] libceph: auth method 'x' error -1

mon.log :

2017-01-21 16:13:53.588022 7f39a0ce2700 1 leveldb: Manual compaction at level-0 
from 'pgmap\x00full_47841' @ 72057594037927935 : 1 .. 'pgmap\x00full_48095' @ 0 
: 0; will stop at (end)

2017-01-21 16:13:53.588046 7f39a0ce2700 1 leveldb: Manual compaction at level-1 
from 'pgmap\x00full_47841' @ 72057594037927935 : 1 .. 'pgmap\x00full_48095' @ 0 
: 0; will stop at (end)

2017-01-21 16:13:53.726893 7f39a6113700 0 cephx server client.admin: unexpected 
key: req.key=2dbc276f1917eca0 expected_key=5b91a0b9b1879ead
2017-01-21 16:13:54.904611 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "quorum_status", "format": "json"} v 0) v1
2017-01-21 16:13:55.777065 7f39ab13c700 0 log_channel(audit) log [DBG] : 
from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-01-21 16:13:58.404417 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "osd tree", "format": "json"} v 0) v1
2017-01-21 16:13:59.573932 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"dumpcontents": ["pools"], "prefix": "pg dump", "format": "json"} 
v 0) v1
2017-01-21 16:14:01.276068 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "config-key get", "key": "handy_key"} v 0) v1
2017-01-21 16:14:01.312275 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "config-key get", "key": "handy_key"} v 0) v1
2017-01-21 16:14:01.916661 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "config-key get", "key": "handy_key"} v 0) v1
2017-01-21 16:14:01.921959 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "config-key get", "key": "handy_key"} v 0) v1
2017-01-21 16:14:02.219041 7f39a6113700 0 
mon.cluster-2@1(peon) e1 handle_command 
mon_command({"prefix": "quorum_status", "format": "json"} v 0) v1
2017-01-21 16:14:03.739007 7f39a6113700 0 cephx server client.admin: unexpected 
key: req.key=1c117aaa13d47454 expected_key=63047567546439d0


Thanks
Best regards

-
本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from H3C, 
which is
intended only for the person or entity whose address is listed above. Any use 
of the
information contained herein in any way (including, but not limited to, total 
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender
by phone or email immediately and delete it!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mon is stuck in leveldb and costs nearly 100% cpu

[Chenyehua]

Dear Mr Kefu Chai
Sorry to disturb you.
I meet a problem recently. In my ceph cluster ,health status has warning “store 
is getting too big!” for several days; and  ceph-mon costs nearly 100% cpu;
Have you ever met this situation?
Some detailed information are attached below:

root@cvknode17:~# ceph -s
cluster 04afba60-3a77-496c-b616-2ecb5e47e141
 health HEALTH_WARN
mon.cvknode17 store is getting too big! 34104 MB >= 15360 MB
 monmap e1: 3 mons at 
{cvknode15=172.16.51.15:6789/0,cvknode16=172.16.51.16:6789/0,cvknode17=172.16.51.17:6789/0}
election epoch 862, quorum 0,1,2 cvknode15,cvknode16,cvknode17
 osdmap e196279: 347 osds: 347 up, 347 in
  pgmap v5891025: 33272 pgs, 16 pools, 26944 GB data, 6822 kobjects
65966 GB used, 579 TB / 644 TB avail
   33270 active+clean
   2 active+clean+scrubbing+deep
  client io 840 kB/s rd, 739 kB/s wr, 35 op/s rd, 184 op/s wr

root@cvknode17:~# top
top - 15:19:28 up 23 days, 23:58,  6 users,  load average: 1.08, 1.40, 1.77
Tasks: 346 total,   2 running, 342 sleeping,   0 stopped,   2 zombie
Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  2.5%si,  0.0%st
Mem:  65384424k total, 58102880k used,  7281544k free,   240720k buffers
Swap: 2100k total,   344944k used, 29654156k free, 24274272k cached

PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
  24407 root  20   0 17.3g  12g  10m S   98 20.2   8420:11 ceph-mon

root@cvknode17:~# top -Hp 24407
top - 15:19:49 up 23 days, 23:59,  6 users,  load average: 1.12, 1.39, 1.76
Tasks:  17 total,   1 running,  16 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.1%us, 10.8%sy,  0.0%ni, 69.0%id,  9.5%wa,  0.0%hi,  2.5%si,  0.0%st
Mem:  65384424k total, 58104868k used,  7279556k free,   240744k buffers
Swap: 2100k total,   344944k used, 29654156k free, 24271188k cached

PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
  25931 root  20   0 17.3g  12g   9m R   98 20.2   7957:37 ceph-mon
  24514 root  20   0 17.3g  12g   9m S2 20.2   3:06.75 ceph-mon
  25932 root  20   0 17.3g  12g   9m S2 20.2   1:07.82 ceph-mon
  24407 root  20   0 17.3g  12g   9m S0 20.2   0:00.67 ceph-mon
  24508 root  20   0 17.3g  12g   9m S0 20.2  15:50.24 ceph-mon
  24513 root  20   0 17.3g  12g   9m S0 20.2   0:07.88 ceph-mon
  24534 root  20   0 17.3g  12g   9m S0 20.2 196:33.85 ceph-mon
  24535 root  20   0 17.3g  12g   9m S0 20.2   0:00.01 ceph-mon
  25929 root  20   0 17.3g  12g   9m S0 20.2   3:06.09 ceph-mon
  25930 root  20   0 17.3g  12g   9m S0 20.2   8:12.58 ceph-mon
  25933 root  20   0 17.3g  12g   9m S0 20.2   4:42.22 ceph-mon
  25934 root  20   0 17.3g  12g   9m S0 20.2  40:53.27 ceph-mon
  25935 root  20   0 17.3g  12g   9m S0 20.2   0:04.84 ceph-mon
  25936 root  20   0 17.3g  12g   9m S0 20.2   0:00.01 ceph-mon
  25980 root  20   0 17.3g  12g   9m S0 20.2   0:06.65 ceph-mon
  25986 root  20   0 17.3g  12g   9m S0 20.2  48:26.77 ceph-mon
  55738 root  20   0 17.3g  12g   9m S0 20.2   0:09.06 ceph-mon


Thread 20 (Thread 0x7f3e77e80700 (LWP 25931)):
#0  0x7f3e7e83a653 in pread64 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x009286cf in ?? ()
#2  0x0092c187 in leveldb::ReadBlock(leveldb::RandomAccessFile*, 
leveldb::ReadOptions const&, leveldb::BlockHandle const&, leveldb::Block**) ()
#3  0x00922f41 in leveldb::Table::BlockReader(void*, 
leveldb::ReadOptions const&, leveldb::Slice const&) ()
#4  0x00924840 in ?? ()
#5  0x00924b39 in ?? ()
#6  0x00924a7a in ?? ()
#7  0x009227d0 in ?? ()
#8  0x009140b6 in ?? ()
#9  0x009143dd in ?? ()
#10 0x0088d399 in 
LevelDBStore::LevelDBWholeSpaceIteratorImpl::lower_bound(std::string const&, 
std::string const&) ()
#11 0x0088bf00 in LevelDBStore::get(std::string const&, 
std::set, std::allocator > 
const&, std::map, 
std::allocator > >*) ()
#12 0x0056a7a2 in MonitorDBStore::get(std::string const&, std::string 
const&) ()
---Type  to continue, or q  to quit---
#13 0x005dcf61 in PaxosService::refresh(bool*) ()
#14 0x0058a76b in Monitor::refresh_from_paxos(bool*) ()
#15 0x005c55ac in Paxos::do_refresh() ()
#16 0x005cc093 in Paxos::handle_commit(MMonPaxos*) ()
#17 0x005d4d8b in Paxos::dispatch(PaxosServiceMessage*) ()
#18 0x005ac204 in Monitor::dispatch(MonSession*, Message*, bool) ()
#19 0x005a9b09 in Monitor::_ms_dispatch(Message*) ()
#20 0x005c48a2 in Monitor::ms_dispatch(Message*) ()
#21 0x008b2e67 in Messenger::ms_deliver_dispatch(Message*) ()
#22 0x008b000a in DispatchQueue::entry() ()
#23 0x007a069d in DispatchQueue::DispatchThread::entry() ()
#24 0x7f3e7e832e9a in start_thread () from 
/lib/x86_64-linux-gnu/libpthread.so.0
#25 0x7f3e7cff63