James,

If your replication factor is 3, for every 1GB added, your GB avail
with decrease by 3GB.


Cary
-Dynamic

On Mon, Dec 18, 2017 at 6:18 PM, James Okken <james.ok...@dialogic.com> wrote:
> Thanks David.
> Thanks again Cary.
>
> If I have
> 682 GB used, 12998 GB / 13680 GB avail,
> then I still need to divide 13680/3 (my replication setting) to get what my 
> total storage really is, right?
>
> Thanks!
>
>
> James Okken
> Lab Manager
> Dialogic Research Inc.
> 4 Gatehall Drive
> Parsippany
> NJ 07054
> USA
>
> Tel:       973 967 5179
> Email:   james.ok...@dialogic.com
> Web:    www.dialogic.com – The Network Fuel Company
>
> This e-mail is intended only for the named recipient(s) and may contain 
> information that is privileged, confidential and/or exempt from disclosure 
> under applicable law. No waiver of privilege, confidence or otherwise is 
> intended by virtue of communication via the internet. Any unauthorized use, 
> dissemination or copying is strictly prohibited. If you have received this 
> e-mail in error, or are not named as a recipient, please immediately notify 
> the sender and destroy all copies of this e-mail.
>
>
> -----Original Message-----
> From: Cary [mailto:dynamic.c...@gmail.com]
> Sent: Friday, December 15, 2017 5:56 PM
> To: David Turner
> Cc: James Okken; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)
>
> James,
>
> You can set these values in ceph.conf.
>
> [global]
> ...
> osd pool default size         = 3
> osd pool default min size  = 2
> ...
>
> New pools that are created will use those values.
>
> If you run a "ceph -s"  and look at the "usage" line, it shows how much space 
> is: 1 used, 2 available, 3 total. ie.
>
> usage:   19465 GB used, 60113 GB / 79578 GB avail
>
> We choose to use Openstack with Ceph in this decade and do the other things, 
> not because they are easy, but because they are hard...;-p
>
>
> Cary
> -Dynamic
>
> On Fri, Dec 15, 2017 at 10:12 PM, David Turner <drakonst...@gmail.com> wrote:
>> In conjunction with increasing the pool size to 3, also increase the
>> pool min_size to 2.  `ceph df` and `ceph osd df` will eventually show
>> the full size in use in your cluster.  In particular the output of
>> `ceph df` with available size in a pool takes into account the pools 
>> replication size.
>> Continue watching ceph -s or ceph -w to see when the backfilling for
>> your change to replication size finishes.
>>
>> On Fri, Dec 15, 2017 at 5:06 PM James Okken <james.ok...@dialogic.com>
>> wrote:
>>>
>>> This whole effort went extremely well, thanks to Cary, and Im not
>>> used to that with CEPH so far. (And openstack ever....) Thank you
>>> Cary.
>>>
>>> Ive upped the replication factor and now I see "replicated size 3" in
>>> each of my pools. Is this the only place to check replication level?
>>> Is there a Global setting or only a setting per Pool?
>>>
>>> ceph osd pool ls detail
>>> pool 0 'rbd' replicated size 3......
>>> pool 1 'images' replicated size 3...
>>> ...
>>>
>>> One last question!
>>> At this replication level how can I tell how much total space I
>>> actually have now?
>>> Do I just 1/3 the Global size?
>>>
>>> ceph df
>>> GLOBAL:
>>>     SIZE       AVAIL      RAW USED     %RAW USED
>>>     13680G     12998G         682G          4.99
>>> POOLS:
>>>     NAME        ID     USED     %USED     MAX AVAIL     OBJECTS
>>>     rbd         0         0         0         6448G           0
>>>     images      1      216G      3.24         6448G       27745
>>>     backups     2         0         0         6448G           0
>>>     volumes     3      117G      1.79         6448G       30441
>>>     compute     4         0         0         6448G           0
>>>
>>> ceph osd df
>>> ID WEIGHT  REWEIGHT SIZE   USE    AVAIL  %USE VAR  PGS
>>>  0 0.81689  1.00000   836G 36549M   800G 4.27 0.86  67
>>>  4 3.70000  1.00000  3723G   170G  3553G 4.58 0.92 270
>>>  1 0.81689  1.00000   836G 49612M   788G 5.79 1.16  56
>>>  5 3.70000  1.00000  3723G   192G  3531G 5.17 1.04 282
>>>  2 0.81689  1.00000   836G 33639M   803G 3.93 0.79  58
>>>  3 3.70000  1.00000  3723G   202G  3521G 5.43 1.09 291
>>>               TOTAL 13680G   682G 12998G 4.99
>>> MIN/MAX VAR: 0.79/1.16  STDDEV: 0.67
>>>
>>> Thanks!
>>>
>>> -----Original Message-----
>>> From: Cary [mailto:dynamic.c...@gmail.com]
>>> Sent: Friday, December 15, 2017 4:05 PM
>>> To: James Okken
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3 server
>>> cluster)
>>>
>>> James,
>>>
>>>  Those errors are normal. Ceph creates the missing files. You can
>>> check "/var/lib/ceph/osd/ceph-6", before and after you run those
>>> commands to see what files are added there.
>>>
>>>  Make sure you get the replication factor set.
>>>
>>>
>>> Cary
>>> -Dynamic
>>>
>>> On Fri, Dec 15, 2017 at 6:11 PM, James Okken
>>> <james.ok...@dialogic.com>
>>> wrote:
>>> > Thanks again Cary,
>>> >
>>> > Yes, once all the backfilling was done I was back to a Healthy cluster.
>>> > I moved on to the same steps for the next server in the cluster, it
>>> > is backfilling now.
>>> > Once that is done I will do the last server in the cluster, and
>>> > then I think I am done!
>>> >
>>> > Just checking on one thing. I get these messages when running this
>>> > command. I assume this is OK, right?
>>> > root@node-54:~# ceph-osd -i 4 --mkfs --mkkey --osd-uuid
>>> > 25c21708-f756-4593-bc9e-c5506622cf07
>>> > 2017-12-15 17:28:22.849534 7fd2f9e928c0 -1 journal FileJournal::_open:
>>> > disabling aio for non-block journal.  Use journal_force_aio to
>>> > force use of aio anyway
>>> > 2017-12-15 17:28:22.855838 7fd2f9e928c0 -1 journal FileJournal::_open:
>>> > disabling aio for non-block journal.  Use journal_force_aio to
>>> > force use of aio anyway
>>> > 2017-12-15 17:28:22.856444 7fd2f9e928c0 -1
>>> > filestore(/var/lib/ceph/osd/ceph-4) could not find
>>> > #-1:7b3f43c4:::osd_superblock:0# in index: (2) No such file or
>>> > directory
>>> > 2017-12-15 17:28:22.893443 7fd2f9e928c0 -1 created object store
>>> > /var/lib/ceph/osd/ceph-4 for osd.4 fsid
>>> > 2b9f7957-d0db-481e-923e-89972f6c594f
>>> > 2017-12-15 17:28:22.893484 7fd2f9e928c0 -1 auth: error reading file:
>>> > /var/lib/ceph/osd/ceph-4/keyring: can't open
>>> > /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory
>>> > 2017-12-15 17:28:22.893662 7fd2f9e928c0 -1 created new key in
>>> > keyring /var/lib/ceph/osd/ceph-4/keyring
>>> >
>>> > thanks
>>> >
>>> > -----Original Message-----
>>> > From: Cary [mailto:dynamic.c...@gmail.com]
>>> > Sent: Thursday, December 14, 2017 7:13 PM
>>> > To: James Okken
>>> > Cc: ceph-users@lists.ceph.com
>>> > Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3
>>> > server
>>> > cluster)
>>> >
>>> > James,
>>> >
>>> >  Usually once the misplaced data has balanced out the cluster
>>> > should reach a healthy state. If you run a "ceph health detail"
>>> > Ceph will show you some more detail about what is happening.  Is
>>> > Ceph still recovering, or has it stalled? has the "objects misplaced 
>>> > (62.511%"
>>> > changed to a lower %?
>>> >
>>> > Cary
>>> > -Dynamic
>>> >
>>> > On Thu, Dec 14, 2017 at 10:52 PM, James Okken
>>> > <james.ok...@dialogic.com>
>>> > wrote:
>>> >> Thanks Cary!
>>> >>
>>> >> Your directions worked on my first sever. (once I found the
>>> >> missing carriage return in your list of commands, the email musta messed 
>>> >> it up.
>>> >>
>>> >> For anyone else:
>>> >> chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 ceph auth add osd.4
>>> >> osd 'allow *' mon 'allow profile osd' -i
>>> >> /etc/ceph/ceph.osd.4.keyring really is 2 commands:
>>> >> chown -R ceph:ceph /var/lib/ceph/osd/ceph-4  and ceph auth add
>>> >> osd.4 osd 'allow *' mon 'allow profile osd' -i
>>> >> /etc/ceph/ceph.osd.4.keyring
>>> >>
>>> >> Cary, what am I looking for in ceph -w and ceph -s to show the
>>> >> status of the data moving?
>>> >> Seems like the data is moving and that I have some issue...
>>> >>
>>> >> root@node-53:~# ceph -w
>>> >>     cluster 2b9f7957-d0db-481e-923e-89972f6c594f
>>> >>      health HEALTH_WARN
>>> >>             176 pgs backfill_wait
>>> >>             1 pgs backfilling
>>> >>             27 pgs degraded
>>> >>             1 pgs recovering
>>> >>             26 pgs recovery_wait
>>> >>             27 pgs stuck degraded
>>> >>             204 pgs stuck unclean
>>> >>             recovery 10322/84644 objects degraded (12.195%)
>>> >>             recovery 52912/84644 objects misplaced (62.511%)
>>> >>      monmap e3: 3 mons at
>>> >> {node-43=192.168.1.7:6789/0,node-44=192.168.1.5:6789/0,node-45=192.168.1.3:6789/0}
>>> >>             election epoch 138, quorum 0,1,2 node-45,node-44,node-43
>>> >>      osdmap e206: 4 osds: 4 up, 4 in; 177 remapped pgs
>>> >>             flags sortbitwise,require_jewel_osds
>>> >>       pgmap v3936175: 512 pgs, 5 pools, 333 GB data, 58184 objects
>>> >>             370 GB used, 5862 GB / 6233 GB avail
>>> >>             10322/84644 objects degraded (12.195%)
>>> >>             52912/84644 objects misplaced (62.511%)
>>> >>                  308 active+clean
>>> >>                  176 active+remapped+wait_backfill
>>> >>                   26 active+recovery_wait+degraded
>>> >>                    1 active+remapped+backfilling
>>> >>                    1 active+recovering+degraded recovery io 100605
>>> >> kB/s, 14 objects/s
>>> >>   client io 0 B/s rd, 92788 B/s wr, 50 op/s rd, 11 op/s wr
>>> >>
>>> >> 2017-12-14 22:45:57.459846 mon.0 [INF] pgmap v3936174: 512 pgs: 1
>>> >> activating, 1 active+recovering+degraded, 26
>>> >> active+recovery_wait+degraded, 1 active+remapped+backfilling, 307
>>> >> active+clean, 176 active+remapped+wait_backfill; 333 GB data, 369
>>> >> active+GB
>>> >> used, 5863 GB / 6233 GB avail; 0 B/s rd, 101107 B/s wr, 19 op/s;
>>> >> 10354/84644 objects degraded (12.232%); 52912/84644 objects
>>> >> misplaced (62.511%); 12224 kB/s, 2 objects/s recovering
>>> >> 2017-12-14 22:45:58.466736 mon.0 [INF] pgmap v3936175: 512 pgs: 1
>>> >> active+recovering+degraded, 26 active+recovery_wait+degraded, 1
>>> >> active+remapped+backfilling, 308 active+clean, 176 wait_backfill;
>>> >> active+remapped+333 GB data, 370 GB used, 5862 GB /
>>> >> 6233 GB avail; 0 B/s rd, 92788 B/s wr, 61 op/s; 10322/84644
>>> >> objects degraded (12.195%); 52912/84644 objects misplaced
>>> >> (62.511%); 100605 kB/s, 14 objects/s recovering
>>> >> 2017-12-14 22:46:00.474335 mon.0 [INF] pgmap v3936176: 512 pgs: 1
>>> >> active+recovering+degraded, 26 active+recovery_wait+degraded, 1
>>> >> active+remapped+backfilling, 308 active+clean, 176 wait_backfill;
>>> >> active+remapped+333 GB data, 370 GB used, 5862 GB /
>>> >> 6233 GB avail; 0 B/s rd, 434 kB/s wr, 45 op/s; 10322/84644 objects
>>> >> degraded (12.195%); 52912/84644 objects misplaced (62.511%); 84234
>>> >> kB/s, 10 objects/s recovering
>>> >> 2017-12-14 22:46:02.482228 mon.0 [INF] pgmap v3936177: 512 pgs: 1
>>> >> active+recovering+degraded, 26 active+recovery_wait+degraded, 1
>>> >> active+remapped+backfilling, 308 active+clean, 176 wait_backfill;
>>> >> active+remapped+333 GB data, 370 GB used, 5862 GB /
>>> >> 6233 GB avail; 0 B/s rd, 334 kB/s wr
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Cary [mailto:dynamic.c...@gmail.com]
>>> >> Sent: Thursday, December 14, 2017 4:21 PM
>>> >> To: James Okken
>>> >> Cc: ceph-users@lists.ceph.com
>>> >> Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3
>>> >> server
>>> >> cluster)
>>> >>
>>> >> Jim,
>>> >>
>>> >> I am not an expert, but I believe I can assist.
>>> >>
>>> >>  Normally you will only have 1 OSD per drive. I have heard
>>> >> discussions about using multiple OSDs per disk, when using SSDs though.
>>> >>
>>> >>  Once your drives have been installed you will have to format
>>> >> them, unless you are using Bluestore. My steps for formatting are below.
>>> >> Replace the sXX with your drive name.
>>> >>
>>> >> parted -a optimal /dev/sXX
>>> >> print
>>> >> mklabel gpt
>>> >> unit mib
>>> >> mkpart OSD4sdd1 1 -1
>>> >> quit
>>> >> mkfs.xfs -f /dev/sXX1
>>> >>
>>> >> # Run blkid, and copy the UUID for the newly formatted drive.
>>> >> blkid
>>> >> # Add the mount point/UUID to fstab. The mount point will be
>>> >> created later.
>>> >> vi /etc/fstab
>>> >> # For example
>>> >> UUID=6386bac4-7fef-3cd2-7d64-13db51d83b12 /var/lib/ceph/osd/ceph-4
>>> >> xfs
>>> >> rw,noatime,inode64,logbufs=8 0 0
>>> >>
>>> >>
>>> >> # You can then add the OSD to the cluster.
>>> >>
>>> >> uuidgen
>>> >> # Replace the UUID below with the UUID that was created with uuidgen.
>>> >> ceph osd create 23e734d7-96d8-4327-a2b9-0fbdc72ed8f1
>>> >>
>>> >> # Notice what number of osd it creates usually the lowest # OSD
>>> >> available.
>>> >>
>>> >> # Add osd.4 to ceph.conf on all Ceph nodes.
>>> >> vi /etc/ceph/ceph.conf
>>> >> ...
>>> >> [osd.4]
>>> >> public addr = 172.1.3.1
>>> >> cluster addr = 10.1.3.1
>>> >> ...
>>> >>
>>> >> # Now add the mount point.
>>> >> mkdir -p /var/lib/ceph/osd/ceph-4
>>> >> chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
>>> >>
>>> >> # The command below mounts everything in fstab.
>>> >> mount -a
>>> >> # The number after -i below needs changed to the correct OSD ID,
>>> >> and the osd-uuid needs to be changed the UUID created with uuidgen above.
>>> >> Your keyring location may be different and need changed as well.
>>> >> ceph-osd -i 4 --mkfs --mkkey --osd-uuid
>>> >> 23e734d7-96d8-4327-a2b9-0fbdc72ed8f1
>>> >> chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 ceph auth add osd.4
>>> >> osd 'allow *' mon 'allow profile osd' -i
>>> >> /etc/ceph/ceph.osd.4.keyring
>>> >>
>>> >> # Add the new OSD to its host in the crush map.
>>> >> ceph osd crush add osd.4 .0 host=YOURhostNAME
>>> >>
>>> >> # Since the weight used in the previous step was .0, you will need
>>> >> to increase it. I use 1 for a 1TB drive and 5 for a 5TB drive. The
>>> >> command below will reweight osd.4 to 1. You may need to slowly ramp up 
>>> >> this number.
>>> >> ie .10 then .20 etc.
>>> >> ceph osd crush reweight osd.4 1
>>> >>
>>> >> You should now be able to start the drive. You can watch the data
>>> >> move to the drive with a ceph -w. Once data has migrated to the
>>> >> drive, start the next.
>>> >>
>>> >> Cary
>>> >> -Dynamic
>>> >>
>>> >> On Thu, Dec 14, 2017 at 5:34 PM, James Okken
>>> >> <james.ok...@dialogic.com>
>>> >> wrote:
>>> >>> Hi all,
>>> >>>
>>> >>> Please let me know if I am missing steps or using the wrong steps
>>> >>>
>>> >>> I'm hoping to expand my small CEPH cluster by adding 4TB hard
>>> >>> drives to each of the 3 servers in the cluster.
>>> >>>
>>> >>> I also need to change my replication factor from 1 to 3.
>>> >>> This is part of an Openstack environment deployed by Fuel and I
>>> >>> had foolishly set my replication factor to 1 in the Fuel settings 
>>> >>> before deploy.
>>> >>> I know this would have been done better at the beginning. I do
>>> >>> want to keep the current cluster and not start over. I know this
>>> >>> is going thrash my cluster for a while replicating, but there isn't too 
>>> >>> much data on it yet.
>>> >>>
>>> >>>
>>> >>> To start I need to safely turn off each CEPH server and add in
>>> >>> the 4TB
>>> >>> drive:
>>> >>> To do that I am going to run:
>>> >>> ceph osd set noout
>>> >>> systemctl stop ceph-osd@1 (or 2 or 3 on the other servers) ceph
>>> >>> osd tree (to verify it is down) poweroff, install the 4TB drive,
>>> >>> bootup again ceph osd unset noout
>>> >>>
>>> >>>
>>> >>>
>>> >>> Next step wouyld be to get CEPH to use the 4TB drives. Each CEPH
>>> >>> server already has a 836GB OSD.
>>> >>>
>>> >>> ceph> osd df
>>> >>> ID WEIGHT  REWEIGHT SIZE  USE  AVAIL %USE  VAR  PGS
>>> >>>  0 0.81689  1.00000  836G 101G  734G 12.16 0.90 167
>>> >>>  1 0.81689  1.00000  836G 115G  721G 13.76 1.02 166
>>> >>>  2 0.81689  1.00000  836G 121G  715G 14.49 1.08 179
>>> >>>               TOTAL 2509G 338G 2171G 13.47 MIN/MAX VAR: 0.90/1.08
>>> >>> STDDEV: 0.97
>>> >>>
>>> >>> ceph> df
>>> >>> GLOBAL:
>>> >>>     SIZE      AVAIL     RAW USED     %RAW USED
>>> >>>     2509G     2171G         338G         13.47
>>> >>> POOLS:
>>> >>>     NAME        ID     USED     %USED     MAX AVAIL     OBJECTS
>>> >>>     rbd         0         0         0         2145G           0
>>> >>>     images      1      216G      9.15         2145G       27745
>>> >>>     backups     2         0         0         2145G           0
>>> >>>     volumes     3      114G      5.07         2145G       29717
>>> >>>     compute     4         0         0         2145G           0
>>> >>>
>>> >>>
>>> >>> Once I get the 4TB drive into each CEPH server should I look to
>>> >>> increasing the current OSD (ie: to 4836GB)?
>>> >>> Or create a second 4000GB OSD on each CEPH server?
>>> >>> If I am going to create a second OSD on each CEPH server I hope
>>> >>> to use this doc:
>>> >>> http://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/
>>> >>>
>>> >>>
>>> >>>
>>> >>> As far as changing the replication factor from 1 to 3:
>>> >>> Here are my pools now:
>>> >>>
>>> >>> ceph osd pool ls detail
>>> >>> pool 0 'rbd' replicated size 1 min_size 1 crush_ruleset 0
>>> >>> object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags
>>> >>> hashpspool stripe_width 0 pool 1 'images' replicated size 1
>>> >>> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num
>>> >>> 64 last_change 116 flags hashpspool stripe_width 0
>>> >>>         removed_snaps [1~3,b~6,12~8,20~2,24~6,2b~8,34~2,37~20]
>>> >>> pool 2 'backups' replicated size 1 min_size 1 crush_ruleset 0
>>> >>> object_hash rjenkins pg_num 64 pgp_num 64 last_change 7 flags
>>> >>> hashpspool stripe_width 0 pool 3 'volumes' replicated size 1
>>> >>> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256
>>> >>> pgp_num 256 last_change 73 flags hashpspool stripe_width 0
>>> >>>         removed_snaps [1~3]
>>> >>> pool 4 'compute' replicated size 1 min_size 1 crush_ruleset 0
>>> >>> object_hash rjenkins pg_num 64 pgp_num 64 last_change 34 flags
>>> >>> hashpspool stripe_width 0
>>> >>>
>>> >>> I plan on using these steps I saw online:
>>> >>> ceph osd pool set rbd size 3
>>> >>> ceph -s  (Verify that replication completes successfully) ceph
>>> >>> osd pool set images size 3 ceph -s ceph osd pool set backups size
>>> >>> 3 ceph -s ceph osd pool set volumes size 3 ceph -s
>>> >>>
>>> >>>
>>> >>> please let me know any advice or better methods...
>>> >>>
>>> >>> thanks
>>> >>>
>>> >>> --Jim
>>> >>>
>>> >>> _______________________________________________
>>> >>> ceph-users mailing list
>>> >>> ceph-users@lists.ceph.com
>>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to