[ceph-users] Re: A couple OSDs not starting after host reboot

2024-04-04 Thread xu chenhui
Hi, Has there been any progress on this issue ? is there quick recover method? I have same problem with you that first 4k block of osd metadata is invalid. It will pay a heavy price to recreate osd. Thanks. ___ ceph-users mailing list --

[ceph-users] Re: Bucket usage per storage classes

2024-04-04 Thread Tobias Urdin
Hello, There is no such usage collected today, see [1] and [2] – where [2] is a specification on how one community member wanted to implement the feature but nobody has put in the work yet that we know of. [1] https://tracker.ceph.com/issues/47342 [2] https://tracker.ceph.com/issues/54972

[ceph-users] Re: Bucket usage per storage classes

2024-04-04 Thread Ondřej Kukla
Let's take for example a situation where I have a standart storage class backed by HDDs and a fast one on SSDs. The user will mix the classes in the bucket and I would like to know how much space Is he taking on the HDDs and how much on the SSDs so I can bill him.In this scenario I don't care that

[ceph-users] Re: Bucket usage per storage classes

2024-04-04 Thread Anthony D'Atri
A bucket may contain objects spread across multiple storage classes, and AIUI the head object is always in the default storage class, so I'm not sure *exactly* what you're after here. > On Apr 4, 2024, at 17:09, Ondřej Kukla wrote: > > Hello, > > I’m playing around with Storage classes in

[ceph-users] Bucket usage per storage classes

2024-04-04 Thread Ondřej Kukla
Hello, I’m playing around with Storage classes in rgw and I’m looking for ways to see per bucket statistics for the diferent storage classes (for billing purposes etc.). I though that I would add another object to the bucket usage response like for multiparts - rgw.multimeta, but it’s

[ceph-users] Re: Cephadm host keeps trying to set osd_memory_target to less than minimum

2024-04-04 Thread Adam King
Sorry to keep asking for more info, but can I also get what `cephadm gather-facts` on that host returns for "memory_total_kb". Might end up creating a unit test out of this case if we have a calculation bug here. On Thu, Apr 4, 2024 at 4:05 PM Mads Aasted wrote: > sorry for the double send,

[ceph-users] Re: Cephadm host keeps trying to set osd_memory_target to less than minimum

2024-04-04 Thread Mads Aasted
sorry for the double send, forgot to hit reply all so it would appear on the page Hi Adam If we multiply by 0.7, and work through the previous example from that number, we would still arrive at roughly 2.5 gb for each osd. And the host in question is trying to set it to less than 500mb. I have

[ceph-users] Re: Upgraded to Quincy 17.2.7: some S3 buckets inaccessible

2024-04-04 Thread Lorenz Bausch
Thank you again Casey for putting us on the right track regarding the changes in multisite resharding support. When going through the various changelogs we didn't pay too much attention to those changes as this cluster doesn't use any multisite features. We have now upgraded to Reef and all

[ceph-users] Re: Slow ops during recovery for RGW index pool only when degraded OSD is primary

2024-04-04 Thread Wesley Dillingham
Initial indication shows "osd_async_recovery_min_cost = 0" to be a huge win here. Some initial thoughts. Were this not for the fact that the index (and other OMAP pools) were isolated to their own OSDs in this cluster this tunable would seemingly cause data/blob objects from data pools to async

[ceph-users] Re: question about rbd_read_from_replica_policy

2024-04-04 Thread Gregory Farnum
On Thu, Apr 4, 2024 at 8:23 AM Anthony D'Atri wrote: > > Network RTT? No, it's sadly not that clever. There's a crush_location configurable that you can set on clients (to a host, or a datacenter, or any other CRUSH bucket), and as long as part of it matches the CRUSH map then it will feed IOs

[ceph-users] Re: RBD image metric

2024-04-04 Thread Anthony D'Atri
Not a requirement but it makes it a LOT faster. > On Apr 4, 2024, at 03:54, Szabo, Istvan (Agoda) > wrote: > > Hi, > > Let's say thin provisioned and no, no fast-diff and object map enabled. As I > see this is a requirements to be able to use "du". > > > Istvan Szabo > Staff Infrastructure

[ceph-users] Re: question about rbd_read_from_replica_policy

2024-04-04 Thread Anthony D'Atri
Network RTT? > On Apr 4, 2024, at 03:44, Noah Elias Feldt wrote: > > Hello, > I have a question about a setting for RBD. > How exactly does "rbd_read_from_replica_policy" with the value "localize" > work? > According to the RBD documentation, read operations will be sent to the > closest OSD

[ceph-users] Re: CEPHADM_HOST_CHECK_FAILED

2024-04-04 Thread Adam King
First, I guess I would make sure that peon7 and peon12 actually could pass the host check (you can run "cephadm check-host" on the host directly if you have a copy of the cephadm binary there) Then I'd try a mgr failover (ceph mgr fail) to clear out any in memory host values cephadm might have and

[ceph-users] Re: purging already destroyed OSD leads to degraded and misplaced objects?

2024-04-04 Thread tobias tempel
Hi Boris, thank you for your answer. What i really did not expect is, that purging an OSD leads to yet another rebalancing, after already having it destroyed(!), after it was taken out and after the rebalancing seemed to have been completed. So with the next host - i have some more to remove

[ceph-users] Re: purging already destroyed OSD leads to degraded and misplaced objects?

2024-04-04 Thread Boris
Hi Tobias, what we usually do, when we want to remove an OSD is to reweight the crush map to 0. This stops the rebalancing after removing the OSD from the crush map. Setting an OSD to out, keeps it weighted in the crush map and when it gets removed, the cluster will rebalance the PGs to reflect

[ceph-users] purging already destroyed OSD leads to degraded and misplaced objects?

2024-04-04 Thread tobias tempel
Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced

[ceph-users] CEPHADM_HOST_CHECK_FAILED

2024-04-04 Thread arnoud
Hi, I’ve added some new nodes to our Ceph cluster. Only did the host add, had not added the OSD’s yet. Due to a configuration error I had to reinstall some of them. But I forgot to remove the nodes from Ceph first. I did a “ceph orch host rm peon7 --offline —force” before re-adding them to the

[ceph-users] Re: Pacific 16.2.15 `osd noin`

2024-04-04 Thread Zakhar Kirpichenko
Thank you, Eugen. This makes sense. /Z On Thu, 4 Apr 2024 at 10:32, Eugen Block wrote: > Hi, > > the noin flag seems to be only applicable to existing OSDs which are > already in the crushmap. It doesn't apply to newly created OSDs, I > could confirm that in a small test cluster with Pacific

[ceph-users] Re: Pacific 16.2.15 `osd noin`

2024-04-04 Thread Zakhar Kirpichenko
Thanks, this is a good suggestion! /Z On Thu, 4 Apr 2024 at 10:29, Janne Johansson wrote: > Den tors 4 apr. 2024 kl 06:11 skrev Zakhar Kirpichenko : > > Any comments regarding `osd noin`, please? > > > > > > I'm adding a few OSDs to an existing cluster, the cluster is running > with > > > `osd

[ceph-users] Re: RBD image metric

2024-04-04 Thread Szabo, Istvan (Agoda)
Hi, Let's say thin provisioned and no, no fast-diff and object map enabled. As I see this is a requirements to be able to use "du". Istvan Szabo Staff Infrastructure Engineer --- Agoda Services Co., Ltd. e:

[ceph-users] question about rbd_read_from_replica_policy

2024-04-04 Thread Noah Elias Feldt
Hello, I have a question about a setting for RBD. How exactly does "rbd_read_from_replica_policy" with the value "localize" work? According to the RBD documentation, read operations will be sent to the closest OSD as determined by the CRUSH map. How does the client know exactly which OSD I am

[ceph-users] Re: Pacific 16.2.15 `osd noin`

2024-04-04 Thread Eugen Block
Hi, the noin flag seems to be only applicable to existing OSDs which are already in the crushmap. It doesn't apply to newly created OSDs, I could confirm that in a small test cluster with Pacific and Reef. I don't have any insights if that is by design or not, I assume it's supposed to

[ceph-users] Re: Pacific 16.2.15 `osd noin`

2024-04-04 Thread Janne Johansson
Den tors 4 apr. 2024 kl 06:11 skrev Zakhar Kirpichenko : > Any comments regarding `osd noin`, please? > > > > I'm adding a few OSDs to an existing cluster, the cluster is running with > > `osd noout,noin`: > > > > cluster: > > id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 > > health: