[ceph-users] Re: CephFS space usage

[Thorne Lawler]

Thanks Igor,

I have tried that, and the number of objects and bytes_used took a long 
time to drop, but they seem to have dropped back to almost the original 
level:


 * Before creating the file:
 o 3885835 objects
 o 45349150134272 bytes_used
 * After creating the file:
 o 3931663 objects
 o 45924147249152 bytes_used
 * Immediately after deleting the file:
 o 3935995 objects
 o 45978612027392 bytes_used
 * Half an hour after deleting the file:
 o 3886013 objects
 o 45349569585152 bytes_used

Unfortunately, this is all production infrastructure, so there is always 
other activity taking place.


What tools are there to visually inspect the object map and see how it 
relates to the filesystem?


On 15/03/2024 7:18 pm, Igor Fedotov wrote:

ceph df detail --format json-pretty

--

Regards,

Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170

_DDNS

/_*Please note:* The information contained in this email message and any 
attached files may be confidential information, and may also be the 
subject of legal professional privilege. _If you are not the intended 
recipient any use, disclosure or copying of this email is unauthorised. 
_If you received this email in error, please notify Discount Domain Name 
Services Pty Ltd on 03 9815 6868 to report this matter and delete all 
copies of this transmission together with any attachments. /

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: activating+undersized+degraded+remapped

[Wesley Dillingham]

You may be suffering from the "crush gives up too soon" situation:

https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-pg/#crush-gives-up-too-soon

You have a 5+3 with only 8 hosts, you may need to increase your crush
tries. See the link for how to fix

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Sun, Mar 17, 2024 at 8:18 AM Joachim Kraftmayer - ceph ambassador <
joachim.kraftma...@clyso.com> wrote:

> also helpful is the output of:
>
> cephpg{poolnum}.{pg-id}query
>
> ___
> ceph ambassador DACH
> ceph consultant since 2012
>
> Clyso GmbH - Premier Ceph Foundation Member
>
> https://www.clyso.com/
>
> Am 16.03.24 um 13:52 schrieb Eugen Block:
> > Yeah, the whole story would help to give better advice. With EC the
> > default min_size is k+1, you could reduce the min_size to 5
> > temporarily, this might bring the PGs back online. But the long term
> > fix is to have all required OSDs up and have enough OSDs to sustain an
> > outage.
> >
> > Zitat von Wesley Dillingham :
> >
> >> Please share "ceph osd tree" and "ceph osd df tree" I suspect you
> >> have not
> >> enough hosts to satisfy the EC
> >>
> >> On Sat, Mar 16, 2024, 8:04 AM Deep Dish  wrote:
> >>
> >>> Hello
> >>>
> >>> I found myself in the following situation:
> >>>
> >>> [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive
> >>>
> >>> pg 4.3d is stuck inactive for 8d, current state
> >>> activating+undersized+degraded+remapped, last acting
> >>> [4,NONE,46,NONE,10,13,NONE,74]
> >>>
> >>> pg 4.6e is stuck inactive for 9d, current state
> >>> activating+undersized+degraded+remapped, last acting
> >>> [NONE,27,77,79,55,48,50,NONE]
> >>>
> >>> pg 4.cb is stuck inactive for 8d, current state
> >>> activating+undersized+degraded+remapped, last acting
> >>> [6,NONE,42,8,60,22,35,45]
> >>>
> >>>
> >>> I have one cephfs with two backing pools -- one for replicated data,
> >>> the
> >>> other for erasure data.  Each pool is mapped to REPLICATED/ vs.
> >>> ERASURE/
> >>> directories on the filesystem.
> >>>
> >>>
> >>> The above pgs. are affecting the ERASURE pool (5+3) backing the
> >>> FS.   How
> >>> can I get ceph to recover these three PGs?
> >>>
> >>>
> >>>
> >>> Thank you.
> >>> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io
> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: activating+undersized+degraded+remapped

[Joachim Kraftmayer - ceph ambassador]

also helpful is the output of:

cephpg{poolnum}.{pg-id}query

___
ceph ambassador DACH
ceph consultant since 2012

Clyso GmbH - Premier Ceph Foundation Member

https://www.clyso.com/

Am 16.03.24 um 13:52 schrieb Eugen Block:
Yeah, the whole story would help to give better advice. With EC the 
default min_size is k+1, you could reduce the min_size to 5 
temporarily, this might bring the PGs back online. But the long term 
fix is to have all required OSDs up and have enough OSDs to sustain an 
outage.


Zitat von Wesley Dillingham :

Please share "ceph osd tree" and "ceph osd df tree" I suspect you 
have not

enough hosts to satisfy the EC

On Sat, Mar 16, 2024, 8:04 AM Deep Dish  wrote:


Hello

I found myself in the following situation:

[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive

    pg 4.3d is stuck inactive for 8d, current state
activating+undersized+degraded+remapped, last acting
[4,NONE,46,NONE,10,13,NONE,74]

    pg 4.6e is stuck inactive for 9d, current state
activating+undersized+degraded+remapped, last acting
[NONE,27,77,79,55,48,50,NONE]

    pg 4.cb is stuck inactive for 8d, current state
activating+undersized+degraded+remapped, last acting
[6,NONE,42,8,60,22,35,45]


I have one cephfs with two backing pools -- one for replicated data, 
the
other for erasure data.  Each pool is mapped to REPLICATED/ vs. 
ERASURE/

directories on the filesystem.


The above pgs. are affecting the ERASURE pool (5+3) backing the 
FS.   How

can I get ceph to recover these three PGs?



Thank you.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io