Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
Now I also know I have too many PGs! It is fairly confusing to talk about PGs on the Pool page, but only vaguely talk about the number of PGs for the cluster. Here are some examples of confusing statements with suggested alternatives from the online docs: http://ceph.com/docs/master/rados/operations/pools/ A typical configuration uses approximately 100 placement groups per OSD to provide optimal balancing without using up too many computing resources. - A typical configuration uses approximately 100 placement groups per OSD for all pools to provide optimal balancing without using up too many computing resources. http://ceph.com/docs/master/rados/operations/placement-groups/ It is mandatory to choose the value of pg_num because it cannot be calculated automatically. Here are a few values commonly used: - It is mandatory to choose the value of pg_num. Because pg_num depends on the planned number of pools in the cluster, it cannot be determined automatically on pool creation. Please use this calculator: http://ceph.com/pgcalc/; Thanks! C. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
That later change would have _increased_ the number of recommended PG, not decreased it. Weird as our Giant health status was ok before upgrading to Hammer… With your cluster 2048 PGs total (all pools combined!) would be the sweet spot, see: http://ceph.com/pgcalc/ http://ceph.com/pgcalc/ Had read this originally when creating the cluster It seems to me that you increased PG counts assuming that the formula is per pool. Well yes maybe, believe we bumped PGs per status complain in Giant mentioned explicit different pool names, eg. too few PGs in pool-name… so we naturally bumped mentioned pools slightly up til next 2-power until health stop complaining and yes we wondered over this relative high number of PGs in total for the cluster, as we initially had read pgcalc and thought we understood this. ceph.com http://ceph.com/ not responsding presently… - are you saying one needs to consider in advance #pools in a cluster and factor this in when calculating the number of PGs? - If so, how to decide which pool gets what #PG, as this is set per pool, especially if one can’t precalc the amount objects ending up in each pool? But yes understand also that more pools means more PGs per OSD, does this imply using different pools to segregate various data f.ex. per application in same cluster is a bad idea? Using pools as sort of name space segregation makes it easy f.e. to remove/migration data per application and thus a handy segregation tool ImHO. - Are the BCP to consolidate data in few pools per cluster? /Steffen___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
On 16/04/2015, at 01.48, Steffen W Sørensen ste...@me.com wrote: Also our calamari web UI won't authenticate anymore, can’t see any issues in any log under /var/log/calamari, any hints on what to look for are appreciated, TIA! Well this morning it will authenticate me, but seems calamari can’t talk to cluster anymore, wondering where to start digging… or will I need to rebuilt newer version to talk with a hammer cluster? # dpkg -l | egrep -i calamari\|ceph ii calamari-clients 1.2.3.1-2-gc1f14b2all Inktank Calamari user interface ii calamari-server1.3-rc-16-g321cd58amd64 Inktank package containing the Calamari management srever Are this version of calamari able to monitor a Hammer cluster like below? ii ceph 0.94.1-1~bpo70+1 amd64 distributed storage and file system ii ceph-common0.94.1-1~bpo70+1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-deploy1.5.23~bpo70+1all Ceph-deploy is an easy to use configuration tool ii ceph-fs-common 0.94.1-1~bpo70+1 amd64 common utilities to mount and interact with a ceph file system ii ceph-fuse 0.94.1-1~bpo70+1 amd64 FUSE-based client for the Ceph distributed file system ii ceph-mds 0.94.1-1~bpo70+1 amd64 metadata server for the ceph distributed file system ii curl 7.29.0-1~bpo70+1.ceph amd64 command line tool for transferring data with URL syntax ii libcephfs1 0.94.1-1~bpo70+1 amd64 Ceph distributed file system client library ii libcurl3:amd64 7.29.0-1~bpo70+1.ceph amd64 easy-to-use client-side URL transfer library (OpenSSL flavour) ii libcurl3-gnutls:amd64 7.29.0-1~bpo70+1.ceph amd64 easy-to-use client-side URL transfer library (GnuTLS flavour) ii libleveldb1:amd64 1.12.0-1~bpo70+1.ceph amd64 fast key-value storage library ii python-ceph0.94.1-1~bpo70+1 amd64 Meta-package for python libraries for the Ceph libraries ii python-cephfs 0.94.1-1~bpo70+1 amd64 Python libraries for the Ceph libcephfs library ii python-rados 0.94.1-1~bpo70+1 amd64 Python libraries for the Ceph librados library ii python-rbd 0.94.1-1~bpo70+1 amd64 Python libraries for the Ceph librbd library TIA /Steffen ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
On 16/04/2015, at 11.09, Christian Balzer ch...@gol.com wrote: On Thu, 16 Apr 2015 10:46:35 +0200 Steffen W Sørensen wrote: That later change would have _increased_ the number of recommended PG, not decreased it. Weird as our Giant health status was ok before upgrading to Hammer… I'm pretty sure the too many check was added around then, and the the too little warning one earlier. Okay, might explain why too many shown up now :) It seems to me that you increased PG counts assuming that the formula is per pool. Well yes maybe, believe we bumped PGs per status complain in Giant mentioned explicit different pool names, eg. too few PGs in pool-name… Probably something like less then 20 PGs or some such, right? Properly yes, at least fewer than what seemed good for proper distribution Your cluster (OSD count) needs (should really, it is not a hard failure but a warning) to be high enough to satisfy the minimum amount of PGs, so (too) many pools with a small cluster will leave you between a rock and hard place. Right, maybe pgcalc should mention/explain a bit on considering #pools ahead as well... - are you saying one needs to consider in advance #pools in a cluster and factor this in when calculating the number of PGs? Yes. Of course the idea is that pools consume space, so if you have many, you also will have more OSDs to spread your PGs around. In this case we wanted to test out radosgw S3 and thus needed to create the required number of pools which increased #PGs But so far not real any data in GW pools as it failed working for our AS3 compatible App. Now we removed those pools again. And are back down to 4 pool. two for ceph FS and two for RBD images, each with 1024 PGs, but still to many PGs, will try to consolidate the two RBD pools into one or two new with fewer PGs… - If so, how to decide which pool gets what #PG, as this is set per pool, especially if one can’t precalc the amount objects ending up in each pool? Dead reckoning. As in, you should have some idea which pool is going to receive how much data. Certainly, but unless you have a large enough cluster and pools that have predictable utilization, fewer pools are the answer. becasuse this makes it easier to match PGs against #OSDs I see It would be nice somehow if #PGs could be decoupled from pools, but then against how to figure out where each pools object are… Just convient to be have all data from a single App in a seperate pool/name space to easily see usage and perform management tasks :/ It is for me, as I have clusters of similar small size and only one type of usage, RBD images. So they have 1 or 2 pools and that's it. This also results in the smoothest data distribution possible of course. Right, thanks 4 sharing! /Steffen ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
On Thu, 16 Apr 2015 10:46:35 +0200 Steffen W Sørensen wrote: That later change would have _increased_ the number of recommended PG, not decreased it. Weird as our Giant health status was ok before upgrading to Hammer… I'm pretty sure the too many check was added around then, and the the too little warning one earlier. With your cluster 2048 PGs total (all pools combined!) would be the sweet spot, see: http://ceph.com/pgcalc/ http://ceph.com/pgcalc/ Had read this originally when creating the cluster It seems to me that you increased PG counts assuming that the formula is per pool. Well yes maybe, believe we bumped PGs per status complain in Giant mentioned explicit different pool names, eg. too few PGs in pool-name… Probably something like less then 20 PGs or some such, right? so we naturally bumped mentioned pools slightly up til next 2-power until health stop complaining and yes we wondered over this relative high number of PGs in total for the cluster, as we initially had read pgcalc and thought we understood this. Your cluster (OSD count) needs (should really, it is not a hard failure but a warning) to be high enough to satisfy the minimum amount of PGs, so (too) many pools with a small cluster will leave you between a rock and hard place. ceph.com http://ceph.com/ not responsding presently… It's being DoS'ed last I heard. - are you saying one needs to consider in advance #pools in a cluster and factor this in when calculating the number of PGs? Yes. Of course the idea is that pools consume space, so if you have many, you also will have more OSDs to spread your PGs around. - If so, how to decide which pool gets what #PG, as this is set per pool, especially if one can’t precalc the amount objects ending up in each pool? Dead reckoning. As in, you should have some idea which pool is going to receive how much data. But yes understand also that more pools means more PGs per OSD, does this imply using different pools to segregate various data f.ex. per application in same cluster is a bad idea? It certainly can be. Using pools as sort of name space segregation makes it easy f.e. to remove/migration data per application and thus a handy segregation tool ImHO. Certainly, but unless you have a large enough cluster and pools that have predictable utilization, fewer pools are the answer. - Are the BCP to consolidate data in few pools per cluster? It is for me, as I have clusters of similar small size and only one type of usage, RBD images. So they have 1 or 2 pools and that's it. This also results in the smoothest data distribution possible of course. Christian /Steffen -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
Also our calamari web UI won't authenticate anymore, can’t see any issues in
any log under /var/log/calamari, any hints on what to look for are appreciated,
TIA!
# dpkg -l | egrep -i calamari\|ceph
ii calamari-clients 1.2.3.1-2-gc1f14b2all
Inktank Calamari user interface
ii calamari-server1.3-rc-16-g321cd58amd64
Inktank package containing the Calamari management srever
ii ceph 0.94.1-1~bpo70+1 amd64
distributed storage and file system
ii ceph-common0.94.1-1~bpo70+1 amd64
common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy1.5.23~bpo70+1all
Ceph-deploy is an easy to use configuration tool
ii ceph-fs-common 0.94.1-1~bpo70+1 amd64
common utilities to mount and interact with a ceph file system
ii ceph-fuse 0.94.1-1~bpo70+1 amd64
FUSE-based client for the Ceph distributed file system
ii ceph-mds 0.94.1-1~bpo70+1 amd64
metadata server for the ceph distributed file system
ii curl 7.29.0-1~bpo70+1.ceph amd64
command line tool for transferring data with URL syntax
ii libcephfs1 0.94.1-1~bpo70+1 amd64
Ceph distributed file system client library
ii libcurl3:amd64 7.29.0-1~bpo70+1.ceph amd64
easy-to-use client-side URL transfer library (OpenSSL flavour)
ii libcurl3-gnutls:amd64 7.29.0-1~bpo70+1.ceph amd64
easy-to-use client-side URL transfer library (GnuTLS flavour)
ii libleveldb1:amd64 1.12.0-1~bpo70+1.ceph amd64
fast key-value storage library
ii python-ceph0.94.1-1~bpo70+1 amd64
Meta-package for python libraries for the Ceph libraries
ii python-cephfs 0.94.1-1~bpo70+1 amd64
Python libraries for the Ceph libcephfs library
ii python-rados 0.94.1-1~bpo70+1 amd64
Python libraries for the Ceph librados library
ii python-rbd 0.94.1-1~bpo70+1 amd64
Python libraries for the Ceph librbd library
On 16/04/2015, at 00.41, Steffen W Sørensen ste...@me.com wrote:
Hi,
Successfully upgrade a small development 4x node Giant 0.87-1 cluster to
Hammer 0.94-1, each node with 6x OSD - 146GB, 19 pools, mainly 2 in usage.
Only minor thing now ceph -s complaining over too may PGs, previously Giant
had complain of too few, so various pools were bumped up till health status
was okay as before upgrading. Admit, that after bumping PGs up in Giant we
had changed pool sizes from 3 to 2 min 1 in fear of perf. when
backfilling/recovering PGs.
# ceph -s
cluster 16fe2dcf-2629-422f-a649-871deba78bcd
health HEALTH_WARN
too many PGs per OSD (1237 max 300)
monmap e29: 3 mons at
{0=10.0.3.4:6789/0,1=10.0.3.2:6789/0,2=10.0.3.1:6789/0}
election epoch 1370, quorum 0,1,2 2,1,0
mdsmap e142: 1/1/1 up {0=2=up:active}, 1 up:standby
osdmap e3483: 24 osds: 24 up, 24 in
pgmap v3719606: 14848 pgs, 19 pools, 530 GB data, 133 kobjects
1055 GB used, 2103 GB / 3159 GB avail
14848 active+clean
Can we just reduce PGs again and should we decrement in minor steps one pool
at a time…
Any thoughts, TIA!
/Steffen
1. restart the monitor daemons on each node
2. then, restart the osd daemons on each node
3. then, restart the mds daemons on each node
4. then, restart the radosgw daemon on each node
Regards.
--
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
Hi,
Successfully upgrade a small development 4x node Giant 0.87-1 cluster to Hammer
0.94-1, each node with 6x OSD - 146GB, 19 pools, mainly 2 in usage.
Only minor thing now ceph -s complaining over too may PGs, previously Giant had
complain of too few, so various pools were bumped up till health status was
okay as before upgrading. Admit, that after bumping PGs up in Giant we had
changed pool sizes from 3 to 2 min 1 in fear of perf. when
backfilling/recovering PGs.
# ceph -s
cluster 16fe2dcf-2629-422f-a649-871deba78bcd
health HEALTH_WARN
too many PGs per OSD (1237 max 300)
monmap e29: 3 mons at
{0=10.0.3.4:6789/0,1=10.0.3.2:6789/0,2=10.0.3.1:6789/0}
election epoch 1370, quorum 0,1,2 2,1,0
mdsmap e142: 1/1/1 up {0=2=up:active}, 1 up:standby
osdmap e3483: 24 osds: 24 up, 24 in
pgmap v3719606: 14848 pgs, 19 pools, 530 GB data, 133 kobjects
1055 GB used, 2103 GB / 3159 GB avail
14848 active+clean
Can we just reduce PGs again and should we decrement in minor steps one pool at
a time…
Any thoughts, TIA!
/Steffen
1. restart the monitor daemons on each node
2. then, restart the osd daemons on each node
3. then, restart the mds daemons on each node
4. then, restart the radosgw daemon on each node
Regards.
--
François Lafont
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Upgrade from Giant 0.87-1 to Hammer 0.94-1
Hello,
On Thu, 16 Apr 2015 00:41:29 +0200 Steffen W Sørensen wrote:
Hi,
Successfully upgrade a small development 4x node Giant 0.87-1 cluster to
Hammer 0.94-1, each node with 6x OSD - 146GB, 19 pools, mainly 2 in
usage. Only minor thing now ceph -s complaining over too may PGs,
previously Giant had complain of too few, so various pools were bumped
up till health status was okay as before upgrading. Admit, that after
bumping PGs up in Giant we had changed pool sizes from 3 to 2 min 1 in
fear of perf. when backfilling/recovering PGs.
That later change would have _increased_ the number of recommended PG, not
decreased it.
With your cluster 2048 PGs total (all pools combined!) would be the sweet
spot, see:
http://ceph.com/pgcalc/
It seems to me that you increased PG counts assuming that the formula is
per pool.
# ceph -s
cluster 16fe2dcf-2629-422f-a649-871deba78bcd
health HEALTH_WARN
too many PGs per OSD (1237 max 300)
monmap e29: 3 mons at
{0=10.0.3.4:6789/0,1=10.0.3.2:6789/0,2=10.0.3.1:6789/0} election epoch
1370, quorum 0,1,2 2,1,0 mdsmap e142: 1/1/1 up {0=2=up:active}, 1
up:standby osdmap e3483: 24 osds: 24 up, 24 in
pgmap v3719606: 14848 pgs, 19 pools, 530 GB data, 133 kobjects
1055 GB used, 2103 GB / 3159 GB avail
14848 active+clean
This is an insanely high PG count for this cluster and is certain to
impact performance and resource requirements (all these PGs need to peer
after all).
Can we just reduce PGs again and should we decrement in minor steps one
pool at a time…
No, as per the documentation you can only increase PGs and PGPs.
So your options are to totally flatten this cluster or if pools with
important data exist to copy them to new, correctly sized, pools and
delete all the oversized ones after that.
Christian
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
