[ceph-users] rbd_mirroring_delete_delay not removing images with snaps
We use the rbd-mirror as a way to migrate volumes between clusters. The process is enable mirroring on the image to migrate, demote on the primary cluster, promote on the secondary cluster, and then disable mirroring on the image. When we started using `rbd_mirroring_delete_delay` so we could retain a backup of the source image, we noticed volumes with unprotected snaps do not get purged from the trash. Previously, the image and all its snaps would be successfully removed after disabling mirroring. I would expect a similar function when using `rbd_mirroring_delete_delay` as well. Is rbd trash just overly cautious here? -- Tyler Brekke Senior Engineer I tbre...@digitalocean.com -- We're Hiring! <https://do.co/careers> | @digitalocean <https://twitter.com/digitalocean> | YouTube <https://www.youtube.com/digitalocean> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Odd 10-minute delay before recovery IO begins
Sounds like your OSDs were down, but not marked out. Recovery will only occur once they are actually marked out. The default mon_osd_down_out_interval is 10 minutes. You can mark them out explicitly with ceph osd out On Mon, Dec 5, 2022 at 2:20 PM Sean Matheny wrote: > Hi all, > > New Quincy cluster here that I'm just running through some benchmarks > against: > > ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy > (stable) > 11 nodes of 24x 18TB HDD OSDs, 2x 2.9TB SSD OSDs > > I'm seeing a delay of almost exactly 10 minutes when I remove an OSD/node > from the cluster until actual recovery IO begins. This is much different > behaviour that what I'm used to in Nautilus previously, where recovery IO > would commence within seconds. Downed OSDs are reflected in ceph health > within a few seconds (as expected), and affected PGs show as undersized a > few seconds later (as expected). I guess this 10-minute delay may even be a > feature-- accidentally rebooting a node before setting recovery flags would > prevent rebalancing, for example. Just thought it was worth asking in case > it's a bug or something to look deeper into. > > I've read through the OSD config and all of my recovery tuneables look ok, > for example: > https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ > > [ceph: root@ /]# ceph config get osd osd_recovery_delay_start > 20.00 > 3[ceph: root@ /]# ceph config get osd osd_recovery_sleep > 40.00 > 5[ceph: root@ /]# ceph config get osd osd_recovery_sleep_hdd > 60.10 > 7[ceph: root@ /]# ceph config get osd osd_recovery_sleep_ssd > 80.00 > 9[ceph: root@ /]# ceph config get osd osd_recovery_sleep_hybrid > 100.025000 > > Thanks in advance. > > Ngā mihi, > > Sean Matheny > HPC Cloud Platform DevOps Lead > New Zealand eScience Infrastructure (NeSI) > > e: sean.math...@nesi.org.nz > > > > _______ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Tyler Brekke Senior Engineer I tbre...@digitalocean.com -- We're Hiring! <https://do.co/careers> | @digitalocean <https://twitter.com/digitalocean> | YouTube <https://www.youtube.com/digitalocean> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: all monitors deleted, state recovered using documentation .. at what point to start osds ?
Hi Shashi, I think you need to have a mgr running to get updated reporting, which would explain the incorrect ceph status output. Since you have a monitor quorum 1 out of 1, you can start up OSDs. but I would recommend getting all your mons/mgrs back up first. On Tue, Nov 8, 2022 at 5:56 PM Shashi Dahal wrote: > Hi, > > Unfortunately, all 3 monitors were lost. > I followed this -> > > https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-mon/#mon-store-recovery-using-osds > and it is in the current state now. > > id: 234c6a96-8101-49d1-b354-1110e759d572 > health: HEALTH_WARN > mon is allowing insecure global_id reclaim > no active mgr > > services: > mon: 1 daemons, quorum mon1 (age 8m) > mgr: no daemons active > osd: 40 osds: 40 up (since 5M), 40 in (since 5M) > > data: > pools: 0 pools, 0 pgs > objects: 0 objects, 0 B > usage: 0 B used, 0 B / 0 B avail > pgs: > > > all OSD daemons are turned off. > > It says all 40 osds are up, but the osd services are down. > If I run ceph osd dump, it shows all the volumes and pg numbers .. so it > looks like it knows of all those. > > My question is, is it now in a state where it's safe to start an OSD daemon > ? Because ceph status shows no pools, no pgs, I have not turned it on to > ensure no data loss occurs. At what point would I start the osd daemon? > > > Note: > If anyone has done something like this before, and can offer (paid) > assistance/consultation, that is also welcome . > > Thanks, > _______ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Tyler Brekke Senior Engineer I tbre...@digitalocean.com -- We're Hiring! <https://do.co/careers> | @digitalocean <https://twitter.com/digitalocean> | YouTube <https://www.youtube.com/digitalocean> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Question about quorum
Hi Murilo, Since we need a majority to maintain a quorum when you lost 2 mons, you only had 50% available and lost quorum. This is why all recommendations specify having an odd number of mons. As you do not get any added availability with 4 instead of 3. If you had 5 mons, you can lose two without losing availability. On Thu, Nov 3, 2022, 2:55 PM Murilo Morais wrote: > Good afternoon everyone! > > I have a lab with 4 mons, I was testing the behavior in case a certain > amount of hosts went offline, as soon as the second one went offline > everything stopped. It would be interesting if there was a fifth node to > ensure that, if two fall, everything will work, but why did everything stop > with only 2 nodes when if there were 3 nodes in the cluster and one fell, > everything would still be working? Is there no way to get this behavior > with 4 nodes? > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: 2-Layer CRUSH Map Rule?
Hi Matthew,
You just have to take two steps when writing your crush rule. First you
want to get 3 different hosts, then you need 2 osd from each host.
ceph osd getcrushmap -o /tmp/crush
crushtool -d /tmp/crush -o /tmp/crush.txt
#edit it / make new rule
rule custom-ec-ruleset {
id 3
type erasure
min_size 4
max_size 6
step take your-root
step choose indep 3 type host
step chooseleaf indep 2 type osd
step emit
}
crushtool -c /tmp/crush.txt -o /tmp/crush.new
You can use the crushtool to test the mappings and make sure they are
working as you expect.
crushtool -i /tmp/crush.new --test --show-mappings --rule 3 --num-rep 6
You can then compare the OSD id and make sure its exactly what you are
looking for.
You can set the crushmap with
ceph osd setcrushmap -i /tmp/crush.new
Note: If you are overwriting your current rule, your data will need to
rebalance as soon as your set the crushmap, close to 100% of your objects
will move. If you create a new rule, you can set your pool to use the new
pool id anytime you are ready.
On Sun, Sep 25, 2022 at 12:49 AM duluxoz wrote:
> Hi Everybody (Hi Dr. Nick),
>
> TL/DR: Is is possible to have a "2-Layer" Crush Map?
>
> I think it is (although I'm not sure about how to set it up).
>
> My issue is that we're using 4-2 Erasure coding on our OSDs, with 7-OSDs
> per OSD-Node (yes, the Cluster is handling things AOK - we're running at
> about 65-70% utilisation of CPU, RAM, etc, so no problem there).
>
> However, we only have 3 OSD-Nodes, and I'd like to ensure that each Node
> has 2 of each pool's OSDs so that if we loose a Node the other 2 can
> take up the slack. I know(?) that with 4-2 EC we can loose 2 out of the
> 6 OSDs, but I'm worried that if we loose a Node it'll take more than 2
> OSDs with it, rendering us "stuffed" (stuffed: a technical term which is
> used as a substitute for a four-letter word rhyming with "truck") 😁
>
> Anyone have any pointers?
>
> Cheers
>
> Dulux-Oz
>
> _______
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
--
Tyler Brekke
Senior Engineer I
tbre...@digitalocean.com
--
We're Hiring! <https://do.co/careers> | @digitalocean
<https://twitter.com/digitalocean> | YouTube
<https://www.youtube.com/digitalocean>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
