[ceph-users] Re: enabling pg_autoscaler on a large production storage?
Those df's and PG numbers all look fine to me. I wouldn't start adjusting pg_num now -- leave the autoscaler module disabled. Some might be concerned about having 190 PGs on an OSD, but this is fine provided you have ample memory (at least 3GB per OSD). Cheers, Dan On Tue, Jun 16, 2020 at 2:23 PM Boris Behrens wrote: > > Oh ok. Because we have two types of ssds. 1.8TB and 3.6TB. > The 1.8TB got around 90-100pgs and the 3.6TB around 150-190pgs > > Here is the output: > RAW STORAGE: > CLASS SIZEAVAIL USEDRAW USED %RAW USED > ssd 956 TiB 360 TiB 595 TiB 596 TiB 62.35 > TOTAL 956 TiB 360 TiB 595 TiB 596 TiB 62.35 > > POOLS: > POOL ID STORED OBJECTS USED%USED MAX > AVAIL > pool 11 580 GiB 149.31k 1.7 TiB 0.5896 > TiB > pool 33 208 TiB 61.66M 593 TiB 67.2196 > TiB > pool 44 7.9 MiB 2.01k 68 MiB 096 > TiB > pool 5519 B 2 36 KiB 096 > TiB > > > > Am Di., 16. Juni 2020 um 14:13 Uhr schrieb Dan van der Ster > : > > > > On Tue, Jun 16, 2020 at 2:00 PM Boris Behrens wrote: > > > > > > See inline comments > > > Dan van der Ster 于2020年6月16日周二 下午7:07写道： > > > > > > > > Could you share the output of > > > > > > > > ceph osd pool ls detail > > > > > > pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash > > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > > > 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width > > > 0 application rbd > > > pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash > > > rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change > > > 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps > > > min_write_recency_for_promote 1 stripe_width 0 application rbd > > > pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash > > > rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 > > > flags hashpspool min_write_recency_for_promote 1 stripe_width 0 > > > application rbd > > > pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash > > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > > > 2318859 flags hashpspool,selfmanaged_snaps > > > min_write_recency_for_promote 1 stripe_width 0 application rbd > > > > > > > OK now maybe share the output of `ceph df` so we can see how much data > > is in each pool? > > > > Assuming that the majority of your data is in 'pool 3' with 16384 PGs, > > then your current PG values are just fine. (You should have around 110 > > PGs per OSD). > > The pg_autoscaler aims for 100 per OSD and doesn't make changes unless > > a pool has 4x too few or too many PGs. > > > > Unless you are planning to put a large proportion of data into the > > other pools, I'd leave pg_autoscaler disabled and move on to the next > > task. > > > > -- Dan > > > > > the mgr module is not enabled yet. > > > > > > > > > > > ? > > > > > > > > This way we can see how the pools are configured and help recommend if > > > > pg_autoscaler is worth enabling. > > > > > > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
Oh ok. Because we have two types of ssds. 1.8TB and 3.6TB. The 1.8TB got around 90-100pgs and the 3.6TB around 150-190pgs Here is the output: RAW STORAGE: CLASS SIZEAVAIL USEDRAW USED %RAW USED ssd 956 TiB 360 TiB 595 TiB 596 TiB 62.35 TOTAL 956 TiB 360 TiB 595 TiB 596 TiB 62.35 POOLS: POOL ID STORED OBJECTS USED%USED MAX AVAIL pool 11 580 GiB 149.31k 1.7 TiB 0.5896 TiB pool 33 208 TiB 61.66M 593 TiB 67.2196 TiB pool 44 7.9 MiB 2.01k 68 MiB 096 TiB pool 5519 B 2 36 KiB 096 TiB Am Di., 16. Juni 2020 um 14:13 Uhr schrieb Dan van der Ster : > > On Tue, Jun 16, 2020 at 2:00 PM Boris Behrens wrote: > > > > See inline comments > > Dan van der Ster 于2020年6月16日周二 下午7:07写道： > > > > > > Could you share the output of > > > > > > ceph osd pool ls detail > > > > pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > > 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width > > 0 application rbd > > pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash > > rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change > > 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps > > min_write_recency_for_promote 1 stripe_width 0 application rbd > > pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash > > rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 > > flags hashpspool min_write_recency_for_promote 1 stripe_width 0 > > application rbd > > pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > > 2318859 flags hashpspool,selfmanaged_snaps > > min_write_recency_for_promote 1 stripe_width 0 application rbd > > > > OK now maybe share the output of `ceph df` so we can see how much data > is in each pool? > > Assuming that the majority of your data is in 'pool 3' with 16384 PGs, > then your current PG values are just fine. (You should have around 110 > PGs per OSD). > The pg_autoscaler aims for 100 per OSD and doesn't make changes unless > a pool has 4x too few or too many PGs. > > Unless you are planning to put a large proportion of data into the > other pools, I'd leave pg_autoscaler disabled and move on to the next > task. > > -- Dan > > > the mgr module is not enabled yet. > > > > > > > > ? > > > > > > This way we can see how the pools are configured and help recommend if > > > pg_autoscaler is worth enabling. > > > > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
On Tue, Jun 16, 2020 at 2:00 PM Boris Behrens wrote: > > See inline comments > > Am Di., 16. Juni 2020 um 13:29 Uhr schrieb Zhenshi Zhou : > > > > I did this on my cluster and there was a huge number of pg rebalanced. > > I think setting this option to 'on' is a good idea if it's a brand new > > cluster. > > > On our new cluster we enabled them, but not on our primary cluster > with the most. > > Dan van der Ster 于2020年6月16日周二 下午7:07写道： > > > > Could you share the output of > > > > ceph osd pool ls detail > > pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width > 0 application rbd > pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change > 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps > min_write_recency_for_promote 1 stripe_width 0 application rbd > pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 > flags hashpspool min_write_recency_for_promote 1 stripe_width 0 > application rbd > pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change > 2318859 flags hashpspool,selfmanaged_snaps > min_write_recency_for_promote 1 stripe_width 0 application rbd > OK now maybe share the output of `ceph df` so we can see how much data is in each pool? Assuming that the majority of your data is in 'pool 3' with 16384 PGs, then your current PG values are just fine. (You should have around 110 PGs per OSD). The pg_autoscaler aims for 100 per OSD and doesn't make changes unless a pool has 4x too few or too many PGs. Unless you are planning to put a large proportion of data into the other pools, I'd leave pg_autoscaler disabled and move on to the next task. -- Dan > the mgr module is not enabled yet. > > > > > ? > > > > This way we can see how the pools are configured and help recommend if > > pg_autoscaler is worth enabling. > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > im groüen Saal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
See inline comments Am Di., 16. Juni 2020 um 13:29 Uhr schrieb Zhenshi Zhou : > > I did this on my cluster and there was a huge number of pg rebalanced. > I think setting this option to 'on' is a good idea if it's a brand new > cluster. > On our new cluster we enabled them, but not on our primary cluster with the most. Dan van der Ster 于2020年6月16日周二 下午7:07写道： > > Could you share the output of > > ceph osd pool ls detail pool 1 'pool 1' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 3 'pool 3' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16384 pgp_num 16384 autoscale_mode warn last_change 2544040 lfor 0/0/1952329 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd pool 4 'pool 4' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode warn last_change 2318859 flags hashpspool min_write_recency_for_promote 1 stripe_width 0 application rbd pool 5 'pool 5' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 2318859 flags hashpspool,selfmanaged_snaps min_write_recency_for_promote 1 stripe_width 0 application rbd the mgr module is not enabled yet. > > ? > > This way we can see how the pools are configured and help recommend if > pg_autoscaler is worth enabling. > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
I did this on my cluster and there was a huge number of pg rebalanced. I think setting this option to 'on' is a good idea if it's a brand new cluster. Dan van der Ster 于2020年6月16日周二 下午7:07写道： > Could you share the output of > > ceph osd pool ls detail > > ? > > This way we can see how the pools are configured and help recommend if > pg_autoscaler is worth enabling. > > Cheers, Dan > > On Tue, Jun 16, 2020 at 11:51 AM Boris Behrens wrote: > > > > I read about the "warm" option and we are already discussing this. > > > > I don't know if the pgs needs a tuning. I don't know what the impact > > is and if there will be any difference if we enable it. > > > > The, meanwhile gone, last ceph admin made a ticket, and I am not > > particularly familiar with ceph. So I need to work on this ticket and > > I try not to trash our ceph storage :-) > > > > Am Di., 16. Juni 2020 um 11:39 Uhr schrieb Dan van der Ster > > : > > > > > > Hi, > > > > > > I agree with "someone" -- it's not a good idea to just naively enable > > > pg_autoscaler on an existing cluster with lots of data and active > > > customers. > > > > > > If you're curious about this feature, it would be harmless to start > > > out by enabling it with pg_autoscale_mode = warn on each pool. > > > This way you can see what the autoscaler would do if it were set to > > > *on*. Then you can tweak all the target_size_ratio or > > > target_size_bytes accordingly. > > > > > > BTW, do you have some feeling that your 17000 PGs are currently not > > > correctly proportioned for your cluster? > > > > > > -- Dan > > > > > > On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens wrote: > > > > > > > > Hi, > > > > > > > > I would like to enable the pg_autoscaler on our nautilus cluster. > > > > Someone told me that I should be really really careful to NOT have > > > > customer impact. > > > > > > > > Maybe someone can share some experience on this? > > > > > > > > The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte > > > > raw storage where ~600TB raw is used. > > > > ___ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > > -- > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > > im groüen Saal. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
Could you share the output of ceph osd pool ls detail ? This way we can see how the pools are configured and help recommend if pg_autoscaler is worth enabling. Cheers, Dan On Tue, Jun 16, 2020 at 11:51 AM Boris Behrens wrote: > > I read about the "warm" option and we are already discussing this. > > I don't know if the pgs needs a tuning. I don't know what the impact > is and if there will be any difference if we enable it. > > The, meanwhile gone, last ceph admin made a ticket, and I am not > particularly familiar with ceph. So I need to work on this ticket and > I try not to trash our ceph storage :-) > > Am Di., 16. Juni 2020 um 11:39 Uhr schrieb Dan van der Ster > : > > > > Hi, > > > > I agree with "someone" -- it's not a good idea to just naively enable > > pg_autoscaler on an existing cluster with lots of data and active > > customers. > > > > If you're curious about this feature, it would be harmless to start > > out by enabling it with pg_autoscale_mode = warn on each pool. > > This way you can see what the autoscaler would do if it were set to > > *on*. Then you can tweak all the target_size_ratio or > > target_size_bytes accordingly. > > > > BTW, do you have some feeling that your 17000 PGs are currently not > > correctly proportioned for your cluster? > > > > -- Dan > > > > On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens wrote: > > > > > > Hi, > > > > > > I would like to enable the pg_autoscaler on our nautilus cluster. > > > Someone told me that I should be really really careful to NOT have > > > customer impact. > > > > > > Maybe someone can share some experience on this? > > > > > > The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte > > > raw storage where ~600TB raw is used. > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend > im groüen Saal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
I read about the "warm" option and we are already discussing this. I don't know if the pgs needs a tuning. I don't know what the impact is and if there will be any difference if we enable it. The, meanwhile gone, last ceph admin made a ticket, and I am not particularly familiar with ceph. So I need to work on this ticket and I try not to trash our ceph storage :-) Am Di., 16. Juni 2020 um 11:39 Uhr schrieb Dan van der Ster : > > Hi, > > I agree with "someone" -- it's not a good idea to just naively enable > pg_autoscaler on an existing cluster with lots of data and active > customers. > > If you're curious about this feature, it would be harmless to start > out by enabling it with pg_autoscale_mode = warn on each pool. > This way you can see what the autoscaler would do if it were set to > *on*. Then you can tweak all the target_size_ratio or > target_size_bytes accordingly. > > BTW, do you have some feeling that your 17000 PGs are currently not > correctly proportioned for your cluster? > > -- Dan > > On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens wrote: > > > > Hi, > > > > I would like to enable the pg_autoscaler on our nautilus cluster. > > Someone told me that I should be really really careful to NOT have > > customer impact. > > > > Maybe someone can share some experience on this? > > > > The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte > > raw storage where ~600TB raw is used. > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: enabling pg_autoscaler on a large production storage?
Hi, I agree with "someone" -- it's not a good idea to just naively enable pg_autoscaler on an existing cluster with lots of data and active customers. If you're curious about this feature, it would be harmless to start out by enabling it with pg_autoscale_mode = warn on each pool. This way you can see what the autoscaler would do if it were set to *on*. Then you can tweak all the target_size_ratio or target_size_bytes accordingly. BTW, do you have some feeling that your 17000 PGs are currently not correctly proportioned for your cluster? -- Dan On Tue, Jun 16, 2020 at 11:31 AM Boris Behrens wrote: > > Hi, > > I would like to enable the pg_autoscaler on our nautilus cluster. > Someone told me that I should be really really careful to NOT have > customer impact. > > Maybe someone can share some experience on this? > > The Cluster got 455 OSDs on 19 hosts with ~17000 PGs and ~1petabyte > raw storage where ~600TB raw is used. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
