Re: [ceph-users] trimming the MON level db

[Luis Periquito]

On Sat, Apr 28, 2018 at 10:24 AM, Wido den Hollander  wrote:
>
>
> On 04/27/2018 08:31 PM, David Turner wrote:
>> I'm assuming that the "very bad move" means that you have some PGs not
>> in active+clean.  Any non-active+clean PG will prevent your mons from
>> being able to compact their db store.  This is by design so that if
>> something were to happen where the data on some of the copies of the PG
>> were lost and gone forever the mons could do their best to enable the
>> cluster to reconstruct the PG knowing when OSDs went down/up, when PGs
>> moved to new locations, etc.
>>
>> Thankfully there isn't a way around this.  Something you can do is stop
>> a mon, move the /var/lib/mon/$(hostname -s)/ folder to a new disk with
>> more space, set it to mount in the proper location, and start it back
>> up.  You would want to do this for each mon to give them more room for
>> the mon store to grow.  Make sure to give the mon plenty of time to get
>> back up into the quorum before moving on to the next one.
>>
>
> Indeed. This is a unknown thing with Monitors for a lot of people. I
> always suggest installing a >200GB DC-grade SSD in Monitors to make sure
> you can make large movements without running into trouble with the MONs.
>
> So yes, move this data to a new disk. Without all PGs active+clean you
> can't trim the store.

It's currently on a P3700 400G SSD and almost full (280g). We've been
able to avoid it to stop growing at such a rate as it was before. The
main issue was with an OSD crashing under load making a few others
crash/OoM as well. Setting nodown has been a saver.
>
>> On Wed, Apr 25, 2018 at 10:25 AM Luis Periquito > > wrote:
>>
>> Hi all,
>>
>> we have a (really) big cluster that's ongoing a very bad move and the
>> monitor database is growing at an alarming rate.
>>
>> The cluster is running jewel (10.2.7) and is there any way to trim the
>> monitor database before it gets HEALTH_OK?
>>
>> I've searched and so far only found people saying not really, but just
>> wanted a final sanity check...
>>
>> thanks,
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] trimming the MON level db

[Wido den Hollander]

On 04/27/2018 08:31 PM, David Turner wrote:
> I'm assuming that the "very bad move" means that you have some PGs not
> in active+clean.  Any non-active+clean PG will prevent your mons from
> being able to compact their db store.  This is by design so that if
> something were to happen where the data on some of the copies of the PG
> were lost and gone forever the mons could do their best to enable the
> cluster to reconstruct the PG knowing when OSDs went down/up, when PGs
> moved to new locations, etc.
> 
> Thankfully there isn't a way around this.  Something you can do is stop
> a mon, move the /var/lib/mon/$(hostname -s)/ folder to a new disk with
> more space, set it to mount in the proper location, and start it back
> up.  You would want to do this for each mon to give them more room for
> the mon store to grow.  Make sure to give the mon plenty of time to get
> back up into the quorum before moving on to the next one.
> 

Indeed. This is a unknown thing with Monitors for a lot of people. I
always suggest installing a >200GB DC-grade SSD in Monitors to make sure
you can make large movements without running into trouble with the MONs.

So yes, move this data to a new disk. Without all PGs active+clean you
can't trim the store.

> On Wed, Apr 25, 2018 at 10:25 AM Luis Periquito  > wrote:
> 
> Hi all,
> 
> we have a (really) big cluster that's ongoing a very bad move and the
> monitor database is growing at an alarming rate.
> 
> The cluster is running jewel (10.2.7) and is there any way to trim the
> monitor database before it gets HEALTH_OK?
> 
> I've searched and so far only found people saying not really, but just
> wanted a final sanity check...
> 
> thanks,
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] trimming the MON level db

[David Turner]

I'm assuming that the "very bad move" means that you have some PGs not in
active+clean.  Any non-active+clean PG will prevent your mons from being
able to compact their db store.  This is by design so that if something
were to happen where the data on some of the copies of the PG were lost and
gone forever the mons could do their best to enable the cluster to
reconstruct the PG knowing when OSDs went down/up, when PGs moved to new
locations, etc.

Thankfully there isn't a way around this.  Something you can do is stop a
mon, move the /var/lib/mon/$(hostname -s)/ folder to a new disk with more
space, set it to mount in the proper location, and start it back up.  You
would want to do this for each mon to give them more room for the mon store
to grow.  Make sure to give the mon plenty of time to get back up into the
quorum before moving on to the next one.

On Wed, Apr 25, 2018 at 10:25 AM Luis Periquito  wrote:

> Hi all,
>
> we have a (really) big cluster that's ongoing a very bad move and the
> monitor database is growing at an alarming rate.
>
> The cluster is running jewel (10.2.7) and is there any way to trim the
> monitor database before it gets HEALTH_OK?
>
> I've searched and so far only found people saying not really, but just
> wanted a final sanity check...
>
> thanks,
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] trimming the MON level db

[Luis Periquito]

Hi all,

we have a (really) big cluster that's ongoing a very bad move and the
monitor database is growing at an alarming rate.

The cluster is running jewel (10.2.7) and is there any way to trim the
monitor database before it gets HEALTH_OK?

I've searched and so far only found people saying not really, but just
wanted a final sanity check...

thanks,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com