[ceph-users] Re: What to expect on rejoining a host to cluster?

[Frank Schilder]

Hi Matt,

that I'm using re-weights does not mean I would recommend it. There seems 
something seriously broken with reweights, see this message 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/E5BYQ27LRWFNT4M34OYKI2KM27Q3DUY6/
 and the thread with it. I have to wait for a client update before considering 
to change that to upmaps.

I'm pretty sure the script https://github.com/TheJJ/ceph-balancer linked by 
Stefan can be configured/tweaked to look only at the fullest and maybe the 
emptiest OSDs to compute a moderately sized list of re-mappings that will 
eliminate the outliers only. What I would not recommend is to go all balanced 
and 95% OSD utilisation. You will see serious performance loss after some OSDs 
reached 80% and if you loose an OSD or host you will have to combat the fallout 
of deleted upmaps.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: What to expect on rejoining a host to cluster?

[Matt Larson]

Frank,

 Then if you have only a few OSDs with excessive PG counts / usage, do you
reweight it down by something like 10-20% to acheive a better distribution
and improve capacity?  Do weight it back to normal after PGs have moved?

 I wondered if manually picking on some of the higher data usage OSDs could
get to a gold outcome and avoid continous rebalancing or other issues.

 Thanks,
   Matt

On Mon, Dec 5, 2022 at 4:32 AM Frank Schilder  wrote:

> Hi Matt,
>
> I can't comment on balancers, I don't use them. I manually re-weight OSDs,
> which fits well with our pools' OSD allocation. Also, we don't aim for
> perfect balance, we just remove the peak of allocation on the fullest few
> OSDs to avoid excessive capacity loss. Not balancing too much has the pro
> of being fairly stable under OSD failures/additions at the expanse of a few
> % less capacity.
>
> Maybe someone else an help here?
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Matt Larson 
> Sent: 04 December 2022 02:00:11
> To: Eneko Lacunza
> Cc: Frank Schilder; ceph-users
> Subject: Re: [ceph-users] Re: What to expect on rejoining a host to
> cluster?
>
> Thank you Frank and Eneko,
>
>  Without help and support from ceph admins like you, I would be adrift.  I
> really appreciate this.
>
>  I rejoined the host now one week ago, and the cluster has been dealing
> with the misplaced objects and recovering well.
>
> I will use this strategy in the future:
>
> "If you consider replacing the host and all disks, get a new host first
> and give it the host name in the crush map. Just before you deploy the new
> host, simply purge all down OSDs in its bucket (set norebalance) and
> deploy. Then, the data movement is restricted to re-balancing to the new
> host.
>
> If you just want to throw out the old host, destroy the OSDs but keep the
> IDs intact (ceph osd destroy). Then, no further re-balancing will happen
> and you can re-use the OSD ids later when adding a new host. That's a
> stable situation from an operations point of view."
>
> Last question I have is that I am now seeing that some OSDs have uneven
> load of PGs, which balancer do you recommend and any caveats for how the
> balancer operations can affect/slow the cluster?
>
> Thanks,
>   Matt
>
> On Mon, Nov 28, 2022 at 2:23 AM Eneko Lacunza  elacu...@binovo.es>> wrote:
> Hi Matt,
>
> Also, make sure that when rejoining host has correct time. I have seen
> clusters going down when rejoining hosts that were down for maintenance for
> various weeks and came in with datetime deltas of some months (no idea why
> that happened, I arrived with the firefighter team ;-) )
>
> Cheers
>
> El 27/11/22 a las 13:27, Frank Schilder escribió:
>
> Hi Matt,
>
> if you didn't touch the OSDs on that host, they will join and only objects
> that have been modified will actually be updated. Ceph keeps some basic
> history information and can detect changes. 2 weeks is not a very long
> time. If you have a lot of cold data, re-integration will go fast.
>
> Initially, you will see a huge amount of misplaced objects. However, this
> count will go down much faster than objects/s recovery.
>
> Before you rejoin the host, I would fix its issues though. Now that you
> have it out of the cluster, do the maintenance first. There is no rush. In
> fact, you can buy a new host, install the OSDs in the new one and join that
> to the cluster with the host-name of the old host.
>
> If you consider replacing the host and all disks, the get a new host first
> and give it the host name in the crush map. Just before you deploy the new
> host, simply purge all down OSDs in its bucket (set norebalance) and
> deploy. Then, the data movement is restricted to re-balancing to the new
> host.
>
> If you just want to throw out the old host, destroy the OSDs but keep the
> IDs intact (ceph osd destroy). Then, no further re-balancing will happen
> and you can re-use the OSD ids later when adding a new host. That's a
> stable situation from an operations point of view.
>
> Hope that helps.
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Matt Larson <mailto:larsonma...@gmail.com>
> Sent: 26 November 2022 21:07:41
> To: ceph-users
> Subject: [ceph-users] What to expect on rejoining a host to cluster?
>
> Hi all,
>
>  I have had a host with 16 OSDs, each 14TB in capacity that started having
> hardware issues causing it to crash.  I took this host down 2 weeks ago,

[ceph-users] Re: What to expect on rejoining a host to cluster?

[Stefan Kooman]

On 12/5/22 10:32, Frank Schilder wrote:

Hi Matt,

I can't comment on balancers, I don't use them. I manually re-weight OSDs, 
which fits well with our pools' OSD allocation. Also, we don't aim for perfect 
balance, we just remove the peak of allocation on the fullest few OSDs to avoid 
excessive capacity loss. Not balancing too much has the pro of being fairly 
stable under OSD failures/additions at the expanse of a few % less capacity.

Maybe someone else an help here?



Try JJ's Ceph balancer [1]. In our case it turned out to be *way* more 
efficient than built-in balancer (faster conversion, less movements 
involved). And able to achieve a very good PG distribution and "reclaim" 
lot's of space. I Highly recommended it.


Gr. Stefan

[1]: https://github.com/TheJJ/ceph-balancer
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: What to expect on rejoining a host to cluster?

[Frank Schilder]

Hi Matt,

I can't comment on balancers, I don't use them. I manually re-weight OSDs, 
which fits well with our pools' OSD allocation. Also, we don't aim for perfect 
balance, we just remove the peak of allocation on the fullest few OSDs to avoid 
excessive capacity loss. Not balancing too much has the pro of being fairly 
stable under OSD failures/additions at the expanse of a few % less capacity.

Maybe someone else an help here?

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Matt Larson 
Sent: 04 December 2022 02:00:11
To: Eneko Lacunza
Cc: Frank Schilder; ceph-users
Subject: Re: [ceph-users] Re: What to expect on rejoining a host to cluster?

Thank you Frank and Eneko,

 Without help and support from ceph admins like you, I would be adrift.  I 
really appreciate this.

 I rejoined the host now one week ago, and the cluster has been dealing with 
the misplaced objects and recovering well.

I will use this strategy in the future:

"If you consider replacing the host and all disks, get a new host first and 
give it the host name in the crush map. Just before you deploy the new host, 
simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, 
the data movement is restricted to re-balancing to the new host.

If you just want to throw out the old host, destroy the OSDs but keep the IDs 
intact (ceph osd destroy). Then, no further re-balancing will happen and you 
can re-use the OSD ids later when adding a new host. That's a stable situation 
from an operations point of view."

Last question I have is that I am now seeing that some OSDs have uneven load of 
PGs, which balancer do you recommend and any caveats for how the balancer 
operations can affect/slow the cluster?

Thanks,
  Matt

On Mon, Nov 28, 2022 at 2:23 AM Eneko Lacunza 
mailto:elacu...@binovo.es>> wrote:
Hi Matt,

Also, make sure that when rejoining host has correct time. I have seen clusters 
going down when rejoining hosts that were down for maintenance for various 
weeks and came in with datetime deltas of some months (no idea why that 
happened, I arrived with the firefighter team ;-) )

Cheers

El 27/11/22 a las 13:27, Frank Schilder escribió:

Hi Matt,

if you didn't touch the OSDs on that host, they will join and only objects that 
have been modified will actually be updated. Ceph keeps some basic history 
information and can detect changes. 2 weeks is not a very long time. If you 
have a lot of cold data, re-integration will go fast.

Initially, you will see a huge amount of misplaced objects. However, this count 
will go down much faster than objects/s recovery.

Before you rejoin the host, I would fix its issues though. Now that you have it 
out of the cluster, do the maintenance first. There is no rush. In fact, you 
can buy a new host, install the OSDs in the new one and join that to the 
cluster with the host-name of the old host.

If you consider replacing the host and all disks, the get a new host first and 
give it the host name in the crush map. Just before you deploy the new host, 
simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, 
the data movement is restricted to re-balancing to the new host.

If you just want to throw out the old host, destroy the OSDs but keep the IDs 
intact (ceph osd destroy). Then, no further re-balancing will happen and you 
can re-use the OSD ids later when adding a new host. That's a stable situation 
from an operations point of view.

Hope that helps.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Matt Larson <mailto:larsonma...@gmail.com>
Sent: 26 November 2022 21:07:41
To: ceph-users
Subject: [ceph-users] What to expect on rejoining a host to cluster?

Hi all,

 I have had a host with 16 OSDs, each 14TB in capacity that started having
hardware issues causing it to crash.  I took this host down 2 weeks ago,
and the data rebalanced to the remaining 11 server hosts in the Ceph
cluster over this time period.

 My initial goal was to then remove the host completely from the cluster
with `ceph osd rm XX` and `ceph osd purge XX` (Adding/Removing OSDs — Ceph
Documentation
<https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/><https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/>).
However, I found that after the large amount of data migration from the
recovery, that the purge and removal from the crush map for an OSDs still
required another large data move.  It appears that it would have been a
better strategy to assign a 0 weight to an OSD to have only a single larger
data move instead of twice.

 I'd like to join the downed server back into the Ceph cluster.  It still
has 14 OSDs that are listed as out/down that would be brought back online.
My question is what can I expect if I bring this h

[ceph-users] Re: What to expect on rejoining a host to cluster?

[Matt Larson]

Thank you Frank and Eneko,

 Without help and support from ceph admins like you, I would be adrift.  I
really appreciate this.

 I rejoined the host now one week ago, and the cluster has been dealing
with the misplaced objects and recovering well.

I will use this strategy in the future:

"If you consider replacing the host and all disks, get a new host first and
give it the host name in the crush map. Just before you deploy the new
host, simply purge all down OSDs in its bucket (set norebalance) and
deploy. Then, the data movement is restricted to re-balancing to the new
host.

If you just want to throw out the old host, destroy the OSDs but keep the
IDs intact (ceph osd destroy). Then, no further re-balancing will happen
and you can re-use the OSD ids later when adding a new host. That's a
stable situation from an operations point of view."

Last question I have is that I am now seeing that some OSDs have uneven
load of PGs, which balancer do you recommend and any caveats for how
the balancer operations can affect/slow the cluster?

Thanks,
  Matt

On Mon, Nov 28, 2022 at 2:23 AM Eneko Lacunza  wrote:

> Hi Matt,
>
> Also, make sure that when rejoining host has correct time. I have seen
> clusters going down when rejoining hosts that were down for maintenance for
> various weeks and came in with datetime deltas of some months (no idea why
> that happened, I arrived with the firefighter team ;-) )
>
> Cheers
>
> El 27/11/22 a las 13:27, Frank Schilder escribió:
>
> Hi Matt,
>
> if you didn't touch the OSDs on that host, they will join and only objects 
> that have been modified will actually be updated. Ceph keeps some basic 
> history information and can detect changes. 2 weeks is not a very long time. 
> If you have a lot of cold data, re-integration will go fast.
>
> Initially, you will see a huge amount of misplaced objects. However, this 
> count will go down much faster than objects/s recovery.
>
> Before you rejoin the host, I would fix its issues though. Now that you have 
> it out of the cluster, do the maintenance first. There is no rush. In fact, 
> you can buy a new host, install the OSDs in the new one and join that to the 
> cluster with the host-name of the old host.
>
> If you consider replacing the host and all disks, the get a new host first 
> and give it the host name in the crush map. Just before you deploy the new 
> host, simply purge all down OSDs in its bucket (set norebalance) and deploy. 
> Then, the data movement is restricted to re-balancing to the new host.
>
> If you just want to throw out the old host, destroy the OSDs but keep the IDs 
> intact (ceph osd destroy). Then, no further re-balancing will happen and you 
> can re-use the OSD ids later when adding a new host. That's a stable 
> situation from an operations point of view.
>
> Hope that helps.
>
> Best regards,
> =
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> 
> From: Matt Larson  
> Sent: 26 November 2022 21:07:41
> To: ceph-users
> Subject: [ceph-users] What to expect on rejoining a host to cluster?
>
> Hi all,
>
>  I have had a host with 16 OSDs, each 14TB in capacity that started having
> hardware issues causing it to crash.  I took this host down 2 weeks ago,
> and the data rebalanced to the remaining 11 server hosts in the Ceph
> cluster over this time period.
>
>  My initial goal was to then remove the host completely from the cluster
> with `ceph osd rm XX` and `ceph osd purge XX` (Adding/Removing OSDs — Ceph
> Documentation
>  ).
> However, I found that after the large amount of data migration from the
> recovery, that the purge and removal from the crush map for an OSDs still
> required another large data move.  It appears that it would have been a
> better strategy to assign a 0 weight to an OSD to have only a single larger
> data move instead of twice.
>
>  I'd like to join the downed server back into the Ceph cluster.  It still
> has 14 OSDs that are listed as out/down that would be brought back online.
> My question is what can I expect if I bring this host online?  Will the
> OSDs of a host that has been offline for an extended period of time and out
> of the cluster have PGs that are now quite different or inconsistent?  Will
> this be problematic?
>
>  Thanks for any advice,
>Matt
>
> --
> Matt Larson, PhD
> Madison, WI  53705 U.S.A.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
>
> Tel. +34 943 569 206 | https://www.binovo.es
> Astigarragako Bide

[ceph-users] Re: What to expect on rejoining a host to cluster?

[Eneko Lacunza]

Hi Matt,

Also, make sure that when rejoining host has correct time. I have seen 
clusters going down when rejoining hosts that were down for maintenance 
for various weeks and came in with datetime deltas of some months (no 
idea why that happened, I arrived with the firefighter team ;-) )


Cheers

El 27/11/22 a las 13:27, Frank Schilder escribió:

Hi Matt,

if you didn't touch the OSDs on that host, they will join and only objects that 
have been modified will actually be updated. Ceph keeps some basic history 
information and can detect changes. 2 weeks is not a very long time. If you 
have a lot of cold data, re-integration will go fast.

Initially, you will see a huge amount of misplaced objects. However, this count 
will go down much faster than objects/s recovery.

Before you rejoin the host, I would fix its issues though. Now that you have it 
out of the cluster, do the maintenance first. There is no rush. In fact, you 
can buy a new host, install the OSDs in the new one and join that to the 
cluster with the host-name of the old host.

If you consider replacing the host and all disks, the get a new host first and 
give it the host name in the crush map. Just before you deploy the new host, 
simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, 
the data movement is restricted to re-balancing to the new host.

If you just want to throw out the old host, destroy the OSDs but keep the IDs 
intact (ceph osd destroy). Then, no further re-balancing will happen and you 
can re-use the OSD ids later when adding a new host. That's a stable situation 
from an operations point of view.

Hope that helps.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Matt Larson
Sent: 26 November 2022 21:07:41
To: ceph-users
Subject: [ceph-users] What to expect on rejoining a host to cluster?

Hi all,

  I have had a host with 16 OSDs, each 14TB in capacity that started having
hardware issues causing it to crash.  I took this host down 2 weeks ago,
and the data rebalanced to the remaining 11 server hosts in the Ceph
cluster over this time period.

  My initial goal was to then remove the host completely from the cluster
with `ceph osd rm XX` and `ceph osd purge XX` (Adding/Removing OSDs — Ceph
Documentation
).
However, I found that after the large amount of data migration from the
recovery, that the purge and removal from the crush map for an OSDs still
required another large data move.  It appears that it would have been a
better strategy to assign a 0 weight to an OSD to have only a single larger
data move instead of twice.

  I'd like to join the downed server back into the Ceph cluster.  It still
has 14 OSDs that are listed as out/down that would be brought back online.
My question is what can I expect if I bring this host online?  Will the
OSDs of a host that has been offline for an extended period of time and out
of the cluster have PGs that are now quite different or inconsistent?  Will
this be problematic?

  Thanks for any advice,
Matt

--
Matt Larson, PhD
Madison, WI  53705 U.S.A.
___
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io
___
ceph-users mailing list --ceph-users@ceph.io
To unsubscribe send an email toceph-users-le...@ceph.io


Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: What to expect on rejoining a host to cluster?

[Frank Schilder]

Hi Matt,

if you didn't touch the OSDs on that host, they will join and only objects that 
have been modified will actually be updated. Ceph keeps some basic history 
information and can detect changes. 2 weeks is not a very long time. If you 
have a lot of cold data, re-integration will go fast.

Initially, you will see a huge amount of misplaced objects. However, this count 
will go down much faster than objects/s recovery.

Before you rejoin the host, I would fix its issues though. Now that you have it 
out of the cluster, do the maintenance first. There is no rush. In fact, you 
can buy a new host, install the OSDs in the new one and join that to the 
cluster with the host-name of the old host.

If you consider replacing the host and all disks, the get a new host first and 
give it the host name in the crush map. Just before you deploy the new host, 
simply purge all down OSDs in its bucket (set norebalance) and deploy. Then, 
the data movement is restricted to re-balancing to the new host.

If you just want to throw out the old host, destroy the OSDs but keep the IDs 
intact (ceph osd destroy). Then, no further re-balancing will happen and you 
can re-use the OSD ids later when adding a new host. That's a stable situation 
from an operations point of view.

Hope that helps.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Matt Larson 
Sent: 26 November 2022 21:07:41
To: ceph-users
Subject: [ceph-users] What to expect on rejoining a host to cluster?

Hi all,

 I have had a host with 16 OSDs, each 14TB in capacity that started having
hardware issues causing it to crash.  I took this host down 2 weeks ago,
and the data rebalanced to the remaining 11 server hosts in the Ceph
cluster over this time period.

 My initial goal was to then remove the host completely from the cluster
with `ceph osd rm XX` and `ceph osd purge XX` (Adding/Removing OSDs — Ceph
Documentation
).
However, I found that after the large amount of data migration from the
recovery, that the purge and removal from the crush map for an OSDs still
required another large data move.  It appears that it would have been a
better strategy to assign a 0 weight to an OSD to have only a single larger
data move instead of twice.

 I'd like to join the downed server back into the Ceph cluster.  It still
has 14 OSDs that are listed as out/down that would be brought back online.
My question is what can I expect if I bring this host online?  Will the
OSDs of a host that has been offline for an extended period of time and out
of the cluster have PGs that are now quite different or inconsistent?  Will
this be problematic?

 Thanks for any advice,
   Matt

--
Matt Larson, PhD
Madison, WI  53705 U.S.A.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io