[ceph-users] Re: A couple OSDs not starting after host reboot

[xu chenhui]

Hi， 
Has there been any progress on this issue ?  is there  quick recover method? I 
have same problem with you that first 4k block of osd metadata is invalid. It 
will pay a heavy price to recreate osd. 

Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bucket usage per storage classes

[Tobias Urdin]

Hello,

There is no such usage collected today, see [1] and [2] – where [2] is a 
specification on how
one community member wanted to implement the feature but nobody has put in the 
work yet
that we know of.

[1] https://tracker.ceph.com/issues/47342
[2] https://tracker.ceph.com/issues/54972

Best regards
Tobias

> On 4 Apr 2024, at 23:59, Ondřej Kukla  wrote:
> 
> Let's take for example a situation where I have a standart storage class 
> backed by HDDs and a fast one on SSDs. The user will mix the classes in the 
> bucket and I would like to know how much space Is he taking on the HDDs and 
> how much on the SSDs so I can bill him.
> 
> In this scenario I don't care that the head object is on HDDs. I would just 
> like to transparently know how much Is stored where.
> 
> I hope it makes sense.
> 
> Ondrej
> 
> 
> On Apr 4, 2024 23:20, Anthony D'Atri  wrote:
> A bucket may contain objects spread across multiple storage classes, and AIUI 
> the head object is always in the default storage class, so I'm not sure 
> *exactly* what you're after here. 
> 
> > On Apr 4, 2024, at 17:09, Ondřej Kukla  wrote: 
> > 
> > Hello, 
> > 
> > I’m playing around with Storage classes in rgw and I’m looking for ways to 
> > see per bucket statistics for the diferent storage classes (for billing 
> > purposes etc.). 
> > 
> > I though that I would add another object to the bucket usage response like 
> > for multiparts - rgw.multimeta, but it’s counted under the rgw.mainIs. 
> > 
> > Is there some option to get this info? 
> > 
> > Ondrej 
> > 
> > 
> > Bucket usage I’m referring to 
> > 
> > "usage": { 
> >"rgw.main": { 
> >"size": 1333179575, 
> >"size_actual": 1333190656, 
> >"size_utilized": 1333179575, 
> >"size_kb": 1301934, 
> >"size_kb_actual": 1301944, 
> >"size_kb_utilized": 1301934, 
> >"num_objects": 4 
> >}, 
> >"rgw.multimeta": { 
> >"size": 0, 
> >"size_actual": 0, 
> >"size_utilized": 0, 
> >"size_kb": 0, 
> >"size_kb_actual": 0, 
> >"size_kb_utilized": 0, 
> >"num_objects": 0 
> >} 
> > } 
> > ___ 
> > ceph-users mailing list -- ceph-users@ceph.io 
> > To unsubscribe send an email to ceph-users-le...@ceph.io 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bucket usage per storage classes

[Ondřej Kukla]

Let's take for example a situation where I have a standart storage class backed by HDDs and a fast one on SSDs. The user will mix the classes in the bucket and I would like to know how much space Is he taking on the HDDs and how much on the SSDs so I can bill him.In this scenario I don't care that the head object is on HDDs. I would just like to transparently know how much Is stored where.I hope it makes sense.OndrejOn Apr 4, 2024 23:20, Anthony D'Atri  wrote:A bucket may contain objects spread across multiple storage classes, and AIUI the head object is always in the default storage class, so I'm not sure *exactly* what you're after here.



> On Apr 4, 2024, at 17:09, Ondřej Kukla  wrote:

> 

> Hello, 

> 

> I’m playing around with Storage classes in rgw and I’m looking for ways to see per bucket statistics for the diferent storage classes (for billing purposes etc.).

> 

> I though that I would add another object to the bucket usage response like for multiparts - rgw.multimeta, but it’s counted under the rgw.mainIs.

> 

> Is there some option to get this info?

> 

> Ondrej

> 

> 

> Bucket usage I’m referring to

> 

> "usage": {

>    "rgw.main": {

>    "size": 1333179575,

>    "size_actual": 1333190656,

>    "size_utilized": 1333179575,

>    "size_kb": 1301934,

>    "size_kb_actual": 1301944,

>    "size_kb_utilized": 1301934,

>    "num_objects": 4

>    },

>    "rgw.multimeta": {

>    "size": 0,

>    "size_actual": 0,

>    "size_utilized": 0,

>    "size_kb": 0,

>    "size_kb_actual": 0,

>    "size_kb_utilized": 0,

>    "num_objects": 0

>    }

> }

> ___

> ceph-users mailing list -- ceph-users@ceph.io

> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Bucket usage per storage classes

[Anthony D'Atri]

A bucket may contain objects spread across multiple storage classes, and AIUI 
the head object is always in the default storage class, so I'm not sure 
*exactly* what you're after here.

> On Apr 4, 2024, at 17:09, Ondřej Kukla  wrote:
> 
> Hello, 
> 
> I’m playing around with Storage classes in rgw and I’m looking for ways to 
> see per bucket statistics for the diferent storage classes (for billing 
> purposes etc.).
> 
> I though that I would add another object to the bucket usage response like 
> for multiparts - rgw.multimeta, but it’s counted under the rgw.mainIs.
> 
> Is there some option to get this info?
> 
> Ondrej
> 
> 
> Bucket usage I’m referring to
> 
> "usage": {
>"rgw.main": {
>"size": 1333179575,
>"size_actual": 1333190656,
>"size_utilized": 1333179575,
>"size_kb": 1301934,
>"size_kb_actual": 1301944,
>"size_kb_utilized": 1301934,
>"num_objects": 4
>},
>"rgw.multimeta": {
>"size": 0,
>"size_actual": 0,
>"size_utilized": 0,
>"size_kb": 0,
>"size_kb_actual": 0,
>"size_kb_utilized": 0,
>"num_objects": 0
>}
> }
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Bucket usage per storage classes

[Ondřej Kukla]

Hello, 

I’m playing around with Storage classes in rgw and I’m looking for ways to see 
per bucket statistics for the diferent storage classes (for billing purposes 
etc.).

I though that I would add another object to the bucket usage response like for 
multiparts - rgw.multimeta, but it’s counted under the rgw.mainIs.

Is there some option to get this info?

Ondrej


Bucket usage I’m referring to

"usage": {
"rgw.main": {
"size": 1333179575,
"size_actual": 1333190656,
"size_utilized": 1333179575,
"size_kb": 1301934,
"size_kb_actual": 1301944,
"size_kb_utilized": 1301934,
"num_objects": 4
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
}
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm host keeps trying to set osd_memory_target to less than minimum

[Adam King]

Sorry to keep asking for more info, but can I also get what `cephadm
gather-facts` on that host returns for "memory_total_kb". Might end up
creating a unit test out of this case if we have a calculation bug here.

On Thu, Apr 4, 2024 at 4:05 PM Mads Aasted  wrote:

> sorry for the double send, forgot to hit reply all so it would appear on
> the page
>
> Hi Adam
>
> If we multiply by 0.7, and work through the previous example from that
> number, we would still arrive at roughly 2.5 gb for each osd. And the host
> in question is trying to set it to less than 500mb.
> I have attached a list of the processes running on the host. Currently you
> can even see that the OSD's are taking up the most memory by far, and at
> least 5x its proposed minimum.
> root@my-ceph01:/# ceph orch ps | grep my-ceph01
> crash.my-ceph01   my-ceph01   running (3w)  7m
> ago  13M9052k-  17.2.6
> grafana.my-ceph01 my-ceph01  *:3000   running (3w)  7m
> ago  13M95.6M-  8.3.5
> mds.testfs.my-ceph01.xjxfzd  my-ceph01   running (3w)  7m
> ago  10M 485M-  17.2.6
> mds.prodfs.my-ceph01.rplvac   my-ceph01   running (3w)  7m
> ago  12M26.9M-  17.2.6
> mds.prodfs.my-ceph01.twikzdmy-ceph01   running (3w)
>  7m ago  12M26.2M-  17.2.6
> mgr.my-ceph01.rxdefe  my-ceph01  *:8443,9283  running (3w)  7m
> ago  13M 907M-  17.2.6
> mon.my-ceph01 my-ceph01   running (3w)  7m
> ago  13M 503M2048M  17.2.6
> node-exporter.my-ceph01   my-ceph01  *:9100   running (3w)  7m
> ago  13M20.4M-  1.5.0
> osd.3my-ceph01   running (3w)
>  7m ago  11M2595M4096M  17.2.6
> osd.5my-ceph01   running (3w)
>  7m ago  11M2494M4096M  17.2.6
> osd.6my-ceph01   running (3w)
>  7m ago  11M2698M4096M  17.2.6
> osd.9my-ceph01   running (3w)
>  7m ago  11M3364M4096M  17.2.6
> prometheus.my-ceph01  my-ceph01  *:9095   running (3w)  7m
> ago  13M 164M-  2.42.0
>
>
>
>
> On Thu, Mar 28, 2024 at 2:13 AM Adam King  wrote:
>
>>  I missed a step in the calculation. The total_memory_kb I mentioned
>> earlier is also multiplied by the value of the
>> mgr/cephadm/autotune_memory_target_ratio before doing the subtractions for
>> all the daemons. That value defaults to 0.7. That might explain it seeming
>> like it's getting a value lower than expected. Beyond that, I'd think 'i'd
>> need a list of the daemon types and count on that host to try and work
>> through what it's doing.
>>
>> On Wed, Mar 27, 2024 at 10:47 AM Mads Aasted  wrote:
>>
>>> Hi Adam.
>>>
>>> So doing the calculations with what you are stating here I arrive at a
>>> total sum for all the listed processes at 13.3 (roughly) gb, for everything
>>> except the osds, leaving well in excess of +4gb for each OSD.
>>> Besides the mon daemon which i can tell on my host has a limit of 2gb ,
>>> none of the other daemons seem to have a limit set according to ceph orch
>>> ps. Then again, they are nowhere near the values stated in min_size_by_type
>>> that you list.
>>> Obviously yes, I could disable the auto tuning, but that would leave me
>>> none the wiser as to why this exact host is trying to do this.
>>>
>>>
>>>
>>> On Tue, Mar 26, 2024 at 10:20 PM Adam King  wrote:
>>>
 For context, the value the autotune goes with takes the value from
 `cephadm gather-facts` on the host (the "memory_total_kb" field) and then
 subtracts from that per daemon on the host according to

 min_size_by_type = {
 'mds': 4096 * 1048576,
 'mgr': 4096 * 1048576,
 'mon': 1024 * 1048576,
 'crash': 128 * 1048576,
 'keepalived': 128 * 1048576,
 'haproxy': 128 * 1048576,
 'nvmeof': 4096 * 1048576,
 }
 default_size = 1024 * 1048576

 what's left is then divided by the number of OSDs on the host to arrive
 at the value. I'll also add, since it seems to be an issue on this
 particular host,  if you add the "_no_autotune_memory" label to the host,
 it will stop trying to do this on that host.

 On Mon, Mar 25, 2024 at 6:32 PM  wrote:

> I have a virtual ceph cluster running 17.2.6 with 4 ubuntu 22.04 hosts
> in it, each with 4 OSD's attached. The first 2 servers hosting mgr's have
> 32GB of RAM each, and the remaining have 24gb
> For some reason i am unable to identify, the first host in the cluster
> appears to constantly be trying to set the osd_memory_target variable to
> roughly half of what the calculated minimum is for the cluster, i see the
> following spamming the logs constantly
> Unable to set osd_memory_target on 

[ceph-users] Re: Cephadm host keeps trying to set osd_memory_target to less than minimum

[Mads Aasted]

sorry for the double send, forgot to hit reply all so it would appear on
the page

Hi Adam

If we multiply by 0.7, and work through the previous example from that
number, we would still arrive at roughly 2.5 gb for each osd. And the host
in question is trying to set it to less than 500mb.
I have attached a list of the processes running on the host. Currently you
can even see that the OSD's are taking up the most memory by far, and at
least 5x its proposed minimum.
root@my-ceph01:/# ceph orch ps | grep my-ceph01
crash.my-ceph01   my-ceph01   running (3w)  7m
ago  13M9052k-  17.2.6
grafana.my-ceph01 my-ceph01  *:3000   running (3w)  7m
ago  13M95.6M-  8.3.5
mds.testfs.my-ceph01.xjxfzd  my-ceph01   running (3w)  7m
ago  10M 485M-  17.2.6
mds.prodfs.my-ceph01.rplvac   my-ceph01   running (3w)  7m
ago  12M26.9M-  17.2.6
mds.prodfs.my-ceph01.twikzdmy-ceph01   running (3w)  7m
ago  12M26.2M-  17.2.6
mgr.my-ceph01.rxdefe  my-ceph01  *:8443,9283  running (3w)  7m
ago  13M 907M-  17.2.6
mon.my-ceph01 my-ceph01   running (3w)  7m
ago  13M 503M2048M  17.2.6
node-exporter.my-ceph01   my-ceph01  *:9100   running (3w)  7m
ago  13M20.4M-  1.5.0
osd.3my-ceph01   running (3w)
 7m ago  11M2595M4096M  17.2.6
osd.5my-ceph01   running (3w)
 7m ago  11M2494M4096M  17.2.6
osd.6my-ceph01   running (3w)
 7m ago  11M2698M4096M  17.2.6
osd.9my-ceph01   running (3w)
 7m ago  11M3364M4096M  17.2.6
prometheus.my-ceph01  my-ceph01  *:9095   running (3w)  7m
ago  13M 164M-  2.42.0




On Thu, Mar 28, 2024 at 2:13 AM Adam King  wrote:

>  I missed a step in the calculation. The total_memory_kb I mentioned
> earlier is also multiplied by the value of the
> mgr/cephadm/autotune_memory_target_ratio before doing the subtractions for
> all the daemons. That value defaults to 0.7. That might explain it seeming
> like it's getting a value lower than expected. Beyond that, I'd think 'i'd
> need a list of the daemon types and count on that host to try and work
> through what it's doing.
>
> On Wed, Mar 27, 2024 at 10:47 AM Mads Aasted  wrote:
>
>> Hi Adam.
>>
>> So doing the calculations with what you are stating here I arrive at a
>> total sum for all the listed processes at 13.3 (roughly) gb, for everything
>> except the osds, leaving well in excess of +4gb for each OSD.
>> Besides the mon daemon which i can tell on my host has a limit of 2gb ,
>> none of the other daemons seem to have a limit set according to ceph orch
>> ps. Then again, they are nowhere near the values stated in min_size_by_type
>> that you list.
>> Obviously yes, I could disable the auto tuning, but that would leave me
>> none the wiser as to why this exact host is trying to do this.
>>
>>
>>
>> On Tue, Mar 26, 2024 at 10:20 PM Adam King  wrote:
>>
>>> For context, the value the autotune goes with takes the value from
>>> `cephadm gather-facts` on the host (the "memory_total_kb" field) and then
>>> subtracts from that per daemon on the host according to
>>>
>>> min_size_by_type = {
>>> 'mds': 4096 * 1048576,
>>> 'mgr': 4096 * 1048576,
>>> 'mon': 1024 * 1048576,
>>> 'crash': 128 * 1048576,
>>> 'keepalived': 128 * 1048576,
>>> 'haproxy': 128 * 1048576,
>>> 'nvmeof': 4096 * 1048576,
>>> }
>>> default_size = 1024 * 1048576
>>>
>>> what's left is then divided by the number of OSDs on the host to arrive
>>> at the value. I'll also add, since it seems to be an issue on this
>>> particular host,  if you add the "_no_autotune_memory" label to the host,
>>> it will stop trying to do this on that host.
>>>
>>> On Mon, Mar 25, 2024 at 6:32 PM  wrote:
>>>
 I have a virtual ceph cluster running 17.2.6 with 4 ubuntu 22.04 hosts
 in it, each with 4 OSD's attached. The first 2 servers hosting mgr's have
 32GB of RAM each, and the remaining have 24gb
 For some reason i am unable to identify, the first host in the cluster
 appears to constantly be trying to set the osd_memory_target variable to
 roughly half of what the calculated minimum is for the cluster, i see the
 following spamming the logs constantly
 Unable to set osd_memory_target on my-ceph01 to 480485376: error
 parsing value: Value '480485376' is below minimum 939524096
 Default is set to 4294967296.
 I did double check and osd_memory_base (805306368) +
 osd_memory_cache_min (134217728) adds up to minimum exactly
 osd_memory_target_autotune is currently enabled. But i cannot for the
 life of me figure out how it is arriving at 480485376 as a value for 

[ceph-users] Re: Upgraded to Quincy 17.2.7: some S3 buckets inaccessible

[Lorenz Bausch]

Thank you again Casey for putting us on the right track regarding the 
changes in multisite resharding support.
When going through the various changelogs we didn't pay too much 
attention to those changes as this cluster doesn't use any multisite 
features.


We have now upgraded to Reef and all buckets except one are usable 
again. The rados objects seem to still exist, so we might have luck with 
something like rgw-restore-bucket-index.


On 03/04/2024 21:43, Casey Bodley wrote:

On Wed, Apr 3, 2024 at 3:09 PM Lorenz Bausch  wrote:


Hi Casey,

thank you so much for analysis! We tested the upgraded intensively, but
the buckets in our test environment were probably too small to get
dynamically resharded.


after upgrading to the Quincy release, rgw would
look at the wrong object names when trying to list those buckets.

As we're currently running Quincy, do you think objects/bucket indexes
might already be altered in a way which makes them also unusable for
Reef?


for multisite resharding support, the bucket instance metadata now
stores an additional 'layout' structure which contains all of the
information necessary to locate its bucket index objects. on reshard,
the Red Hat pacific release would have stored that information with
the bucket. the upstream Reef release should be able to interpret that
layout data correctly

however, if the Quincy release overwrites that bucket instance
metadata (via an operation like PutBucketAcl, PutBucketPolicy, etc),
the corresponding layout information would be erased such that an
upgrade to Reef would not be able to find the real bucket index shard
objects



Kind regards,
Lorenz
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Slow ops during recovery for RGW index pool only when degraded OSD is primary

[Wesley Dillingham]

Initial indication shows "osd_async_recovery_min_cost = 0"  to be a huge
win here. Some initial thoughts. Were this not for the fact that the index
(and other OMAP pools) were isolated to their own OSDs in this cluster this
tunable would seemingly cause data/blob objects from data pools to async
recover when synchronous recovery might be better for those pools / that
data. I can play around with how this affects the RGW data pools. There was
a Ceph code walk thru video of this topic:
https://www.youtube.com/watch?v=waOtatCpnYs it seems that perhaps
osd_async_recovery_min_cost may have previously been referred to as
osd_async_recover_min_pg_log_entries (both default to 100). For a pool with
OMAP data where each or some OMAP objects are very very large this may not
be a dynamic enough factor to base the decision on. Thanks for the feedback
everybody!


Respectfully,

*Wes Dillingham*
LinkedIn 
w...@wesdillingham.com




On Wed, Apr 3, 2024 at 1:38 PM Joshua Baergen 
wrote:

> We've had success using osd_async_recovery_min_cost=0 to drastically
> reduce slow ops during index recovery.
>
> Josh
>
> On Wed, Apr 3, 2024 at 11:29 AM Wesley Dillingham 
> wrote:
> >
> > I am fighting an issue on an 18.2.0 cluster where a restart of an OSD
> which
> > supports the RGW index pool causes crippling slow ops. If the OSD is
> marked
> > with primary-affinity of 0 prior to the OSD restart no slow ops are
> > observed. If the OSD has a primary affinity of 1 slow ops occur. The slow
> > ops only occur during the recovery period of the OMAP data and further
> only
> > occur when client activity is allowed to pass to the cluster. Luckily I
> am
> > able to test this during periods when I can disable all client activity
> at
> > the upstream proxy.
> >
> > Given the behavior of the primary affinity changes preventing the slow
> ops
> > I think this may be a case of recovery being more detrimental than
> > backfill. I am thinking that causing an pg_temp acting set by forcing
> > backfill may be the right method to mitigate the issue. [1]
> >
> > I believe that reducing the PG log entries for these OSDs would
> accomplish
> > that but I am also thinking a tuning of osd_async_recovery_min_cost [2]
> may
> > also accomplish something similar. Not sure the appropriate tuning for
> that
> > config at this point or if there may be a better approach. Seeking any
> > input here.
> >
> > Further if this issue sounds familiar or sounds like another condition
> > within the OSD may be at hand I would be interested in hearing your input
> > or thoughts. Thanks!
> >
> > [1] https://docs.ceph.com/en/latest/dev/peering/#concepts
> > [2]
> >
> https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#confval-osd_async_recovery_min_cost
> >
> > Respectfully,
> >
> > *Wes Dillingham*
> > LinkedIn 
> > w...@wesdillingham.com
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: question about rbd_read_from_replica_policy

[Gregory Farnum]

On Thu, Apr 4, 2024 at 8:23 AM Anthony D'Atri  wrote:
>
> Network RTT?

No, it's sadly not that clever. There's a crush_location configurable
that you can set on clients (to a host, or a datacenter, or any other
CRUSH bucket), and as long as part of it matches the CRUSH map then it
will feed IOs to OSDs within that CRUSH domain.
-Greg

>
> > On Apr 4, 2024, at 03:44, Noah Elias Feldt  wrote:
> >
> > Hello,
> > I have a question about a setting for RBD.
> > How exactly does "rbd_read_from_replica_policy" with the value "localize" 
> > work?
> > According to the RBD documentation, read operations will be sent to the 
> > closest OSD as determined by the CRUSH map. How does the client know 
> > exactly which OSD I am closest to?
> > The client is not in the CRUSH map. I can't find much more information 
> > about it. How does this work?
> >
> > Thanks
> >
> >
> > noah feldt
> > infrastrutur
> > _
> >
> >  > OWAPstImg85013815377039-2715-4125-ae27-665645666b62.png>
> >
> > Mittwald CM Service GmbH & Co. KG
> > Königsberger Straße 4-6
> > 32339 Espelkamp
> >
> > Tel.: 05772 / 293-100
> > Fax: 05772 / 293-333
> >
> > supp...@mittwald.de 
> > https://www.mittwald.de 
> >
> > Geschäftsführer: Robert Meyer, Florian Jürgens
> >
> > USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
> > Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
> >
> > Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit
> > gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds 
> >  abrufbar.
> >
> >
> >
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io 
> > To unsubscribe send an email to ceph-users-le...@ceph.io 
> > 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD image metric

[Anthony D'Atri]

Not a requirement but it makes it a LOT faster.

> On Apr 4, 2024, at 03:54, Szabo, Istvan (Agoda)  
> wrote:
> 
> Hi,
> 
> Let's say thin provisioned and no, no fast-diff and object map enabled. As I 
> see this is a requirements to be able to use "du".
> 
> 
> Istvan Szabo
> Staff Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
> 
> 
> 
> 
> From: Anthony D'Atri 
> Sent: Thursday, April 4, 2024 3:19 AM
> To: Szabo, Istvan (Agoda) 
> Cc: Ceph Users 
> Subject: [ceph-users] Re: RBD image metric
> 
> Email received from the internet. If in doubt, don't click any link nor open 
> any attachment !
> 
> 
> Depending on your Ceph release you might need to enable rbdstats.
> 
> Are you after provisioned, allocated, or both sizes?  Do you have object-map 
> and fast-diff enabled?  They speed up `rbd du` massively.
> 
>> On Apr 3, 2024, at 00:26, Szabo, Istvan (Agoda)  
>> wrote:
>> 
>> Hi,
>> 
>> Trying to pull out some metrics from ceph about the rbd images sizes but 
>> haven't found anything only pool related metrics.
>> 
>> Wonder is there any metric about images or I need to create by myself to 
>> collect it with some third party tool?
>> 
>> Thank you
>> 
>> 
>> This message is confidential and is for the sole use of the intended 
>> recipient(s). It may also be privileged or otherwise protected by copyright 
>> or other legal rules. If you have received it by mistake please let us know 
>> by reply email and delete it from your system. It is prohibited to copy this 
>> message or disclose its content to anyone. Any confidentiality or privilege 
>> is not waived or lost by any mistaken delivery or unauthorized disclosure of 
>> the message. All messages sent to and from Agoda may be monitored to ensure 
>> compliance with company policies, to protect the company's interests and to 
>> remove potential malware. Electronic messages may be intercepted, amended, 
>> lost or deleted, or contain viruses.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: question about rbd_read_from_replica_policy

[Anthony D'Atri]

Network RTT?

> On Apr 4, 2024, at 03:44, Noah Elias Feldt  wrote:
> 
> Hello,
> I have a question about a setting for RBD.
> How exactly does "rbd_read_from_replica_policy" with the value "localize" 
> work?
> According to the RBD documentation, read operations will be sent to the 
> closest OSD as determined by the CRUSH map. How does the client know exactly 
> which OSD I am closest to?
> The client is not in the CRUSH map. I can't find much more information about 
> it. How does this work?
> 
> Thanks
> 
> 
> noah feldt
> infrastrutur
> _
> 
>  OWAPstImg85013815377039-2715-4125-ae27-665645666b62.png>
> 
> Mittwald CM Service GmbH & Co. KG
> Königsberger Straße 4-6
> 32339 Espelkamp
>  
> Tel.: 05772 / 293-100
> Fax: 05772 / 293-333
>  
> supp...@mittwald.de 
> https://www.mittwald.de 
>  
> Geschäftsführer: Robert Meyer, Florian Jürgens
>  
> USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
> Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
> 
> Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit 
> gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds 
>  abrufbar.
> 
> 
> 
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io 
> To unsubscribe send an email to ceph-users-le...@ceph.io 
> 
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CEPHADM_HOST_CHECK_FAILED

[Adam King]

First, I guess I would make sure that peon7 and peon12 actually could pass
the host check (you can run "cephadm check-host" on the host directly if
you have a copy of the cephadm binary there) Then I'd try a mgr failover
(ceph mgr fail) to clear out any in memory host values cephadm might have
and restart the module. If it still reproduces after that, then you might
have to set mgr/cephadm/log_to_cluster_level to debug, do another mgr
failover, wait until the module crashes and see if "ceph log last 100 debug
cephadm" gives more info on where the crash occurred (it might have an
actual traceback).

On Thu, Apr 4, 2024 at 4:51 AM  wrote:

> Hi,
>
> I’ve added some new nodes to our Ceph cluster. Only did the host add, had
> not added the OSD’s yet.
> Due to a configuration error I had to reinstall some of them. But I forgot
> to remove the nodes from Ceph first. I did a “ceph orch host rm peon7
> --offline —force” before re-adding them to the cluster.
>
> All the nodes are showing up in the host list (all the peons are the new
> ones):
>
> # ceph orch host ls
> HOST ADDR LABELS  STATUS
> ceph110.103.0.71
> ceph210.103.0.72
> ceph310.103.0.73
> ceph410.103.0.74
> compute1 10.103.0.11
> compute2 10.103.0.12
> compute3 10.103.0.13
> compute4 10.103.0.14
> controller1  10.103.0.8
> controller2  10.103.0.9
> controller3  10.103.0.10
> peon110.103.0.41
> peon210.103.0.42
> peon310.103.0.43
> peon410.103.0.44
> peon510.103.0.45
> peon610.103.0.46
> peon710.103.0.47
> peon810.103.0.48
> peon910.103.0.49
> peon10   10.103.0.50
> peon12   10.103.0.52
> peon13   10.103.0.53
> peon14   10.103.0.54
> peon15   10.103.0.55
> peon16   10.103.0.56
>
> But Ceph status still shows an error, which I can’t seem to get rid off.
>
> [WRN] CEPHADM_HOST_CHECK_FAILED: 2 hosts fail cephadm check
> host peon7 (10.103.0.47) failed check: Can't communicate with remote
> host `10.103.0.47`, possibly because python3 is not installed there or you
> are missing NOPASSWD in sudoers. [Errno 113] Connect call failed
> ('10.103.0.47', 22)
> host peon12 (10.103.0.52) failed check: Can't communicate with remote
> host `10.103.0.52`, possibly because python3 is not installed there or you
> are missing NOPASSWD in sudoers. [Errno 113] Connect call failed
> ('10.103.0.52', 22)
> [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'peon7'
> Module 'cephadm' has failed: ‘peon7'
>
> From the mgr log:
>
> Apr 04 08:33:46 controller2 bash[4031857]: debug
> 2024-04-04T08:33:46.876+ 7f2bb5710700 -1 mgr.server reply reply (5)
> Input/output error Module 'cephadm' has experienced an error and cannot
> handle commands: 'peon7'
>
> Any idea how to clear this error?
>
> # ceph --version
> ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus
> (stable)
>
>
> Regards,
> Arnoud de Jonge.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: purging already destroyed OSD leads to degraded and misplaced objects?

[tobias tempel]

Hi Boris, 
thank you for your answer. 
What i really did not expect is, that purging an OSD leads to yet another 
rebalancing, after already having it destroyed(!), after it was taken out and 
after the rebalancing seemed to have been completed. 
So with the next host - i have some more to remove - i'll do as you suggest. 
Thanks again, 
cheers, toBias 


From: "Boris"  
To: "Tobias Tempel"  
Cc: "ceph-users"  
Sent: Thursday, 4 April, 2024 11:02:09 
Subject: Re: [ceph-users] purging already destroyed OSD leads to degraded and 
misplaced objects? 

Hi Tobias, 

what we usually do, when we want to remove an OSD is to reweight the 
crush map to 0. This stops the rebalancing after removing the OSD from 
the crush map. Setting an OSD to out, keeps it weighted in the crush 
map and when it gets removed, the cluster will rebalance the PGs to 
reflect the new crush map. 

Am Do., 4. Apr. 2024 um 10:54 Uhr schrieb tobias tempel 
: 
> 
> Dear Cephers, 
> 
> reorganizing one of our clusters i'm removing some hosts from it, taking 
> "out" all OSDs on these hosts and waiting until all PGs are fine. 
> After stopping and destroying all OSDs on one host i notice, that "purge" of 
> such destroyed OSDs temporarily leads to degraded and misplaced objects 
> (tried "reweight 0" as well, same picture) . 
> Why that? I would - due to my limited experience - expect, that completely 
> removing an already destroyed OSD could not have any effect on object 
> placement at all. 
> 
> This effect yet seems not to be a real problem, as recovery works fine ... 
> but perhaps it indicates some other issue. 
> All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's 
> some work to do. 
> 
> Perhaps you can give me a hint, where to look, for an explanation? 
> 
> Thank you 
> cheers, toBias 

-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend 
im groüen Saal. 

-- 
Tobias Tempel 
Deutsches Elektronen-Synchrotron - IT 
Notkestr. 85, 22607 Hamburg, Germany 
email: tobias.tem...@desy.de 


smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: purging already destroyed OSD leads to degraded and misplaced objects?

[Boris]

Hi Tobias,

what we usually do, when we want to remove an OSD is to reweight the
crush map to 0. This stops the rebalancing after removing the OSD from
the crush map. Setting an OSD to out, keeps it weighted in the crush
map and when it gets removed, the cluster will rebalance the PGs to
reflect the new crush map.

Am Do., 4. Apr. 2024 um 10:54 Uhr schrieb tobias tempel :
>
> Dear Cephers,
>
> reorganizing one of our clusters i'm removing some hosts from it, taking 
> "out" all OSDs on these hosts and waiting until all PGs are fine.
> After stopping and destroying all OSDs on one host i notice, that "purge" of 
> such destroyed OSDs temporarily leads to degraded and misplaced objects 
> (tried "reweight 0" as well, same picture) .
> Why that? I would - due to my limited experience - expect, that completely 
> removing an already destroyed OSD could not have any effect on object 
> placement at all.
>
> This effect yet seems not to be a real problem, as recovery works fine ... 
> but perhaps it indicates some other issue.
> All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's 
> some work to do.
>
> Perhaps you can give me a hint, where to look, for an explanation?
>
> Thank you
> cheers, toBias
>
> --
> Tobias Tempel
> Deutsches Elektronen-Synchrotron - IT
> Notkestr. 85, 22607 Hamburg, Germany
> email: tobias.tem...@desy.de
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend
im groüen Saal.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] purging already destroyed OSD leads to degraded and misplaced objects?

[tobias tempel]

Dear Cephers, 

reorganizing one of our clusters i'm removing some hosts from it, taking "out" 
all OSDs on these hosts and waiting until all PGs are fine. 
After stopping and destroying all OSDs on one host i notice, that "purge" of 
such destroyed OSDs temporarily leads to degraded and misplaced objects (tried 
"reweight 0" as well, same picture) . 
Why that? I would - due to my limited experience - expect, that completely 
removing an already destroyed OSD could not have any effect on object placement 
at all. 

This effect yet seems not to be a real problem, as recovery works fine ... but 
perhaps it indicates some other issue. 
All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's 
some work to do. 

Perhaps you can give me a hint, where to look, for an explanation? 

Thank you 
cheers, toBias 

-- 
Tobias Tempel 
Deutsches Elektronen-Synchrotron - IT 
Notkestr. 85, 22607 Hamburg, Germany 
email: tobias.tem...@desy.de 



smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] CEPHADM_HOST_CHECK_FAILED

[arnoud]

Hi,

I’ve added some new nodes to our Ceph cluster. Only did the host add, had not 
added the OSD’s yet.
Due to a configuration error I had to reinstall some of them. But I forgot to 
remove the nodes from Ceph first. I did a “ceph orch host rm peon7 --offline 
—force” before re-adding them to the cluster.

All the nodes are showing up in the host list (all the peons are the new ones):

# ceph orch host ls
HOST ADDR LABELS  STATUS
ceph110.103.0.71
ceph210.103.0.72
ceph310.103.0.73
ceph410.103.0.74
compute1 10.103.0.11
compute2 10.103.0.12
compute3 10.103.0.13
compute4 10.103.0.14
controller1  10.103.0.8
controller2  10.103.0.9
controller3  10.103.0.10
peon110.103.0.41
peon210.103.0.42
peon310.103.0.43
peon410.103.0.44
peon510.103.0.45
peon610.103.0.46
peon710.103.0.47
peon810.103.0.48
peon910.103.0.49
peon10   10.103.0.50
peon12   10.103.0.52
peon13   10.103.0.53
peon14   10.103.0.54
peon15   10.103.0.55
peon16   10.103.0.56

But Ceph status still shows an error, which I can’t seem to get rid off.

[WRN] CEPHADM_HOST_CHECK_FAILED: 2 hosts fail cephadm check
host peon7 (10.103.0.47) failed check: Can't communicate with remote host 
`10.103.0.47`, possibly because python3 is not installed there or you are 
missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.47', 22)
host peon12 (10.103.0.52) failed check: Can't communicate with remote host 
`10.103.0.52`, possibly because python3 is not installed there or you are 
missing NOPASSWD in sudoers. [Errno 113] Connect call failed ('10.103.0.52', 22)
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'peon7'
Module 'cephadm' has failed: ‘peon7'

From the mgr log:

Apr 04 08:33:46 controller2 bash[4031857]: debug 2024-04-04T08:33:46.876+ 
7f2bb5710700 -1 mgr.server reply reply (5) Input/output error Module 'cephadm' 
has experienced an error and cannot handle commands: 'peon7'

Any idea how to clear this error?

# ceph --version
ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable)


Regards,
Arnoud de Jonge.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.15 `osd noin`

[Zakhar Kirpichenko]

Thank you, Eugen. This makes sense.

/Z

On Thu, 4 Apr 2024 at 10:32, Eugen Block  wrote:

> Hi,
>
> the noin flag seems to be only applicable to existing OSDs which are
> already in the crushmap. It doesn't apply to newly created OSDs, I
> could confirm that in a small test cluster with Pacific and Reef. I
> don't have any insights if that is by design or not, I assume it's
> supposed to work like that.
> If you want to prevent data movement when creating new OSDs you could
> use the osd_crush_initial_weight config option and set it to 0. We
> have that in our cluster as well, but of course you'd have to reweight
> new OSDs manually.
>
> Regards,
> Eugen
>
> Zitat von Zakhar Kirpichenko :
>
> > Any comments regarding `osd noin`, please?
> >
> > /Z
> >
> > On Tue, 2 Apr 2024 at 16:09, Zakhar Kirpichenko 
> wrote:
> >
> >> Hi,
> >>
> >> I'm adding a few OSDs to an existing cluster, the cluster is running
> with
> >> `osd noout,noin`:
> >>
> >>   cluster:
> >> id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86
> >> health: HEALTH_WARN
> >> noout,noin flag(s) set
> >>
> >> Specifically `noin` is documented as "prevents booting OSDs from being
> >> marked in". But freshly added OSDs were immediately marked `up` and
> `in`:
> >>
> >>   services:
> >> ...
> >> osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs
> >>  flags noout,noin
> >>
> >> # ceph osd tree in | grep -E "osd.11|osd.12|osd.26"
> >>  11hdd9.38680  osd.11   up   1.0
> 1.0
> >>  12hdd9.38680  osd.12   up   1.0
> 1.0
> >>  26hdd9.38680  osd.26   up   1.0
> 1.0
> >>
> >> Is this expected behavior? Do I misunderstand the purpose of the `noin`
> >> option?
> >>
> >> Best regards,
> >> Zakhar
> >>
> >>
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.15 `osd noin`

[Zakhar Kirpichenko]

Thanks, this is a good suggestion!

/Z

On Thu, 4 Apr 2024 at 10:29, Janne Johansson  wrote:

> Den tors 4 apr. 2024 kl 06:11 skrev Zakhar Kirpichenko :
> > Any comments regarding `osd noin`, please?
> > >
> > > I'm adding a few OSDs to an existing cluster, the cluster is running
> with
> > > `osd noout,noin`:
> > >
> > >   cluster:
> > > id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86
> > > health: HEALTH_WARN
> > > noout,noin flag(s) set
> > >
> > > Specifically `noin` is documented as "prevents booting OSDs from being
> > > marked in". But freshly added OSDs were immediately marked `up` and
> `in`:
>
> Only that we mostly set "initial osd crush weight" to 0.0001 so they
> are up and in at first start, but don't receive PGs, instead of using
> "noin" when adding new OSDs.
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD image metric

[Szabo, Istvan (Agoda)]

Hi,

Let's say thin provisioned and no, no fast-diff and object map enabled. As I 
see this is a requirements to be able to use "du".


Istvan Szabo
Staff Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---




From: Anthony D'Atri 
Sent: Thursday, April 4, 2024 3:19 AM
To: Szabo, Istvan (Agoda) 
Cc: Ceph Users 
Subject: [ceph-users] Re: RBD image metric

Email received from the internet. If in doubt, don't click any link nor open 
any attachment !


Depending on your Ceph release you might need to enable rbdstats.

Are you after provisioned, allocated, or both sizes?  Do you have object-map 
and fast-diff enabled?  They speed up `rbd du` massively.

> On Apr 3, 2024, at 00:26, Szabo, Istvan (Agoda)  
> wrote:
>
> Hi,
>
> Trying to pull out some metrics from ceph about the rbd images sizes but 
> haven't found anything only pool related metrics.
>
> Wonder is there any metric about images or I need to create by myself to 
> collect it with some third party tool?
>
> Thank you
>
> 
> This message is confidential and is for the sole use of the intended 
> recipient(s). It may also be privileged or otherwise protected by copyright 
> or other legal rules. If you have received it by mistake please let us know 
> by reply email and delete it from your system. It is prohibited to copy this 
> message or disclose its content to anyone. Any confidentiality or privilege 
> is not waived or lost by any mistaken delivery or unauthorized disclosure of 
> the message. All messages sent to and from Agoda may be monitored to ensure 
> compliance with company policies, to protect the company's interests and to 
> remove potential malware. Electronic messages may be intercepted, amended, 
> lost or deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] question about rbd_read_from_replica_policy

[Noah Elias Feldt]

Hello,

I have a question about a setting for RBD.

How exactly does "rbd_read_from_replica_policy" with the value "localize" work?

According to the RBD documentation, read operations will be sent to the closest 
OSD as determined by the CRUSH map. How does the client know exactly which OSD 
I am closest to?

The client is not in the CRUSH map. I can't find much more information about 
it. How does this work?


Thanks



noah feldt

infrastrutur

_


[Inline image OWAPstImg850138]


Mittwald CM Service GmbH & Co. KG

Königsberger Straße 4-6

32339 Espelkamp



Tel.: 05772 / 293-100

Fax: 05772 / 293-333



supp...@mittwald.de

https://www.mittwald.de



Geschäftsführer: Robert Meyer, Florian Jürgens



USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen

Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen


Informationen zur Datenverarbeitung im Rahmen unserer Geschäftstätigkeit
gemäß Art. 13-14 DSGVO sind unter www.mittwald.de/ds 
abrufbar.






___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.15 `osd noin`

[Eugen Block]

Hi,

the noin flag seems to be only applicable to existing OSDs which are  
already in the crushmap. It doesn't apply to newly created OSDs, I  
could confirm that in a small test cluster with Pacific and Reef. I  
don't have any insights if that is by design or not, I assume it's  
supposed to work like that.
If you want to prevent data movement when creating new OSDs you could  
use the osd_crush_initial_weight config option and set it to 0. We  
have that in our cluster as well, but of course you'd have to reweight  
new OSDs manually.


Regards,
Eugen

Zitat von Zakhar Kirpichenko :


Any comments regarding `osd noin`, please?

/Z

On Tue, 2 Apr 2024 at 16:09, Zakhar Kirpichenko  wrote:


Hi,

I'm adding a few OSDs to an existing cluster, the cluster is running with
`osd noout,noin`:

  cluster:
id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86
health: HEALTH_WARN
noout,noin flag(s) set

Specifically `noin` is documented as "prevents booting OSDs from being
marked in". But freshly added OSDs were immediately marked `up` and `in`:

  services:
...
osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs
 flags noout,noin

# ceph osd tree in | grep -E "osd.11|osd.12|osd.26"
 11hdd9.38680  osd.11   up   1.0  1.0
 12hdd9.38680  osd.12   up   1.0  1.0
 26hdd9.38680  osd.26   up   1.0  1.0

Is this expected behavior? Do I misunderstand the purpose of the `noin`
option?

Best regards,
Zakhar



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific 16.2.15 `osd noin`

[Janne Johansson]

Den tors 4 apr. 2024 kl 06:11 skrev Zakhar Kirpichenko :
> Any comments regarding `osd noin`, please?
> >
> > I'm adding a few OSDs to an existing cluster, the cluster is running with
> > `osd noout,noin`:
> >
> >   cluster:
> > id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86
> > health: HEALTH_WARN
> > noout,noin flag(s) set
> >
> > Specifically `noin` is documented as "prevents booting OSDs from being
> > marked in". But freshly added OSDs were immediately marked `up` and `in`:

Only that we mostly set "initial osd crush weight" to 0.0001 so they
are up and in at first start, but don't receive PGs, instead of using
"noin" when adding new OSDs.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io