Hello,
On Mon, 13 Mar 2017 21:32:45 +0000 James Okken wrote:
> Hi all,
>
> I have a 3 storage node openstack setup using CEPH.
> I believe that means I have 3 OSDs, as each storage node has a one of 3 fiber
> channel storage locations mounted.
You use "believe" a lot, so I'm assuming you're quite new and unfamiliar
with Ceph.
Reading docs and google are your friends.
> The storage media behind each node is actually single 7TB HP fiber channel
> MSA array.
> The best performance configuration for the hard drives in the MSA just
> happened to be 3x 2.3TB RAID10's. And that matched nicely to the
> 3xStorageNode/OSD of the CEPH setup.
A pretty unusual approach, not that others (including me) have done similar
things.
Having just 3 OSDs is iffy, since there are corner cases where Ceph may
not be able to distribute PGs (using default parameters) with such a small
pool of OSDs.
> I believe my replication factor is 3.
>
You answered that yourself in the dump, a
"ceph osd pool ls detail"
would have been more elegant and yielded the same information.
You also have a min_size of 1, which can be problematic (search the
archives), but with a really small cluster that may still be advantageous.
Lastly, since your OSDs are RAIDs and thus very reliable, a replication of
2 is feasible.
> My question is how much total CEPH storage does this allow me? Only 2.3TB? or
> does the way CEPH duplicates data enable more than 1/3 of the storage?
3 means 3, so 2.3TB. Note that Ceph is spare, so that can help quite a bit.
> A follow up question would be what is the best way to tell, thru CEPH, the
> space used and space free? Thanks!!
>
Well, what do you use with *ix, "df", don'tcha?
"ceph df {detail}"
Christian
> root@node-1:/var/log# ceph osd tree
> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 6.53998 root default
> -5 2.17999 host node-28
> 3 2.17999 osd.3 up 1.00000 1.00000
> -6 2.17999 host node-30
> 4 2.17999 osd.4 up 1.00000 1.00000
> -7 2.17999 host node-31
> 5 2.17999 osd.5 up 1.00000 1.00000
> 0 0 osd.0 down 0 1.00000
> 1 0 osd.1 down 0 1.00000
> 2 0 osd.2 down 0 1.00000
>
>
>
> ##
> root@node-1:/var/log# ceph osd lspools
> 0 rbd,2 volumes,3 backups,4 .rgw.root,5 .rgw.control,6 .rgw,7 .rgw.gc,8
> .users.uid,9 .users,10 compute,11 images,
>
>
>
> ##
> root@node-1:/var/log# ceph osd dump
> epoch 216
> fsid d06d61b0-1cd0-4e1a-ac20-67972d0e1fde
> created 2016-10-11 14:15:05.638099
> modified 2017-03-09 14:45:01.030678
> flags
> pool 0 'rbd' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0
> pool 2 'volumes' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 130 flags hashpspool stripe_width 0
> removed_snaps [1~5]
> pool 3 'backups' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 14 flags hashpspool stripe_width 0
> pool 4 '.rgw.root' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 16 flags hashpspool stripe_width 0
> pool 5 '.rgw.control' replicated size 3 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 64 pgp_num 64 last_change 18 owner
> 18446744073709551615 flags hashpspool stripe_width 0
> pool 6 '.rgw' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 20 owner 18446744073709551615 flags
> hashpspool stripe_width 0
> pool 7 '.rgw.gc' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 21 flags hashpspool stripe_width 0
> pool 8 '.users.uid' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 22 owner 18446744073709551615 flags
> hashpspool stripe_width 0
> pool 9 '.users' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 24 flags hashpspool stripe_width 0
> pool 10 'compute' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 216 flags hashpspool stripe_width 0
> removed_snaps [1~37]
> pool 11 'images' replicated size 3 min_size 1 crush_ruleset 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 189 flags hashpspool stripe_width 0
> removed_snaps [1~3,5~8,f~4,14~2,18~2,1c~1,1e~1]
> max_osd 6
> osd.0 down out weight 0 up_from 48 up_thru 50 down_at 52 last_clean_interval
> [44,45) 192.168.0.9:6800/4485 192.168.1.4:6800/4485 192.168.1.4:6801/4485
> 192.168.0.9:6801/4485 exists,new
> osd.1 down out weight 0 up_from 10 up_thru 48 down_at 50 last_clean_interval
> [5,8) 192.168.0.7:6800/60912 192.168.1.6:6801/60912 192.168.1.6:6802/60912
> 192.168.0.7:6801/60912 exists,new
> osd.2 down out weight 0 up_from 10 up_thru 48 down_at 50 last_clean_interval
> [5,8) 192.168.0.6:6800/61013 192.168.1.7:6800/61013 192.168.1.7:6801/61013
> 192.168.0.6:6801/61013 exists,new
> osd.3 up in weight 1 up_from 192 up_thru 201 down_at 190
> last_clean_interval [83,191) 192.168.0.9:6800/2634194
> 192.168.1.7:6802/3634194 192.168.1.7:6803/3634194 192.168.0.9:6802/3634194
> exists,up 28b02052-3196-4203-bec8-ac83a69fcbc5
> osd.4 up in weight 1 up_from 196 up_thru 201 down_at 194
> last_clean_interval [80,195) 192.168.0.7:6800/2629319
> 192.168.1.6:6802/3629319 192.168.1.6:6803/3629319 192.168.0.7:6802/3629319
> exists,up 124b58e6-1e38-4246-8838-cfc3b88e8a5a
> osd.5 up in weight 1 up_from 201 up_thru 201 down_at 199
> last_clean_interval [134,200) 192.168.0.6:6800/5494 192.168.1.4:6802/1005494
> 192.168.1.4:6803/1005494 192.168.0.6:6802/1005494 exists,up
> ddfca14e-e6f6-4c48-aa8f-0ebfc765d32f
> root@node-1:/var/log#
>
>
> James Okken
> Lab Manager
> Dialogic Research Inc.
> 4 Gatehall Drive
> Parsippany
> NJ 07054
> USA
>
> Tel: 973 967 5179
> Email: james.ok...@dialogic.com<mailto:james.ok...@dialogic.com>
> Web: www.dialogic.com<http://www.dialogic.com/> - The Network Fuel
> Company<http://www.dialogic.com/en/landing/itw.aspx>
>
> This e-mail is intended only for the named recipient(s) and may contain
> information that is privileged, confidential and/or exempt from disclosure
> under applicable law. No waiver of privilege, confidence or otherwise is
> intended by virtue of communication via the internet. Any unauthorized use,
> dissemination or copying is strictly prohibited. If you have received this
> e-mail in error, or are not named as a recipient, please immediately notify
> the sender and destroy all copies of this e-mail.
>
--
Christian Balzer Network/Systems Engineer
ch...@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
