-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I'm not sure why you have six monitors. Six monitors buys you nothing over five monitors other than more power being used, and more latency and more headache. See http://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/#monitor-quorum for some more info. Also, I'd consider 5 monitors overkill for this size cluster, I'd recommend three.
Although this is most likely not the root cause of your problem, you probably have an error here: "root replicated-T1" is pointing to b02s08 and b02s12 and "site erbus" is also pointing to b02s08 and b02s12. You probably meant to have "root replicated-T1" pointing to erbus instead. Where I think your problem is, is in your "rule replicated" section. You can try: step take replicated-T1 step choose firstn 2 type host step chooseleaf firstn 2 type osdgroup step emit What this does is choose two hosts from the root replicated-T1 (which happens to be both hosts you have), then chooses an OSD from two osdgroups on each host. I believe the problem with your current rule set is that firstn 0 type host tries to select four hosts, but only two are available. You should be able to see that with 'ceph pg dump', where only two osds will be listed in the up set. I hope that helps. -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.3.3 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWnR9kCRDmVDuy+mK58QAA5hUP/iJprG4nGR2sJvL//8l+ V6oLYXTCs8lHeKL3ZPagThE9oh2xDMV37WR3I/xMNTA8735grl8/AAhy8ypW MDOikbpzfWnlaL0SWs5rIQ5umATwv73Fg/Mf+K2Olt8IGP6D0NMIxfeOjU6E 0Sc3F37nDQFuDEkBYjcVcqZC89PByh7yaId+eOgr7Ot+BZL/3fbpWIZ9kyD5 KoPYdPjtFruoIpc8DJydzbWdmha65DkB65QOZlI3F3lMc6LGXUopm4OP4sQd txVKFtTcLh97WgUshQMSWIiJiQT7+3D6EqQyPzlnei3O3gACpkpsmUteDPpn p8CDeJtIpgKnQZjBwfK/bUQXdIGem8Y0x/PC+1ekIhkHCIJeW2sD3mFJduDQ 9loQ9+IsWHfQmEHLMLdeNzRXbgBY2djxP2X70fXTg31fx+dYvbWeulYJHiKi 1fJS4GdbPjoRUp5k4lthk3hDTFD/f5ZuowLDIaexgISb0bIJcObEn9RWlHut IRVi0fUuRVIX3snGMOKjLmSUe87Od2KSEbULYPTLYDMo/FsWXWHNlP3gVKKd lQJdxcwXOW7/v5oayY4wiEE6NF4rCupcqt0nPxxmbehmeRPxgkWCKJJs3FNr VmUdnrdpfxzR5c8dmOELJnpNS6MTT56B8A4kKmqbbHCEKpZ83piG7uwqc+6f RKkQ =gp/0 -----END PGP SIGNATURE----- ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sun, Jan 17, 2016 at 6:31 PM, deeepdish <deeepd...@gmail.com> wrote: > Hi Everyone, > > Looking for a double check of my logic and crush map.. > > Overview: > > - osdgroup bucket type defines failure domain within a host of 5 OSDs + 1 > SSD. Therefore 5 OSDs (all utilizing the same journal) constitute an > osdgroup bucket. Each host has 4 osdgroups. > - 6 monitors > - Two node cluster > - Each node: > - 20 OSDs > - 4 SSDs > - 4 osdgroups > > Desired Crush Rule outcome: > - Assuming a pool with min_size=2 and size=4, all each node would contain a > redundant copy of each object. Should any of the hosts fail, access to > data would be uninterrupted. > > Current Crush Rule outcome: > - There are 4 copies of each object, however I don’t believe each node has a > redundant copy of each object, when a node fails, data is NOT accessible > until ceph rebuilds itself / node becomes accessible again. > > I susepct my crush is not right, and to remedy it may take some time and > cause cluster to be unresponsive / unavailable. Is there a way / method > to apply substantial crush changes gradually to a cluster? > > Thanks for your help. > > > Current crush map: > > # begin crush map > tunable choose_local_tries 0 > tunable choose_local_fallback_tries 0 > tunable choose_total_tries 50 > tunable chooseleaf_descend_once 1 > tunable straw_calc_version 1 > > # devices > device 0 osd.0 > device 1 osd.1 > device 2 osd.2 > device 3 osd.3 > device 4 osd.4 > device 5 osd.5 > device 6 osd.6 > device 7 osd.7 > device 8 osd.8 > device 9 osd.9 > device 10 osd.10 > device 11 osd.11 > device 12 osd.12 > device 13 osd.13 > device 14 osd.14 > device 15 osd.15 > device 16 osd.16 > device 17 osd.17 > device 18 osd.18 > device 19 osd.19 > device 20 osd.20 > device 21 osd.21 > device 22 osd.22 > device 23 osd.23 > device 24 osd.24 > device 25 osd.25 > device 26 osd.26 > device 27 osd.27 > device 28 osd.28 > device 29 osd.29 > device 30 osd.30 > device 31 osd.31 > device 32 osd.32 > device 33 osd.33 > device 34 osd.34 > device 35 osd.35 > device 36 osd.36 > device 37 osd.37 > device 38 osd.38 > device 39 osd.39 > > # types > type 0 osd > type 1 osdgroup > type 2 host > type 3 rack > type 4 site > type 5 root > > # buckets > osdgroup b02s08-osdgroupA { > id -81 # do not change unnecessarily > # weight 18.100 > alg straw > hash 0 # rjenkins1 > item osd.0 weight 3.620 > item osd.1 weight 3.620 > item osd.2 weight 3.620 > item osd.3 weight 3.620 > item osd.4 weight 3.620 > } > osdgroup b02s08-osdgroupB { > id -82 # do not change unnecessarily > # weight 18.100 > alg straw > hash 0 # rjenkins1 > item osd.5 weight 3.620 > item osd.6 weight 3.620 > item osd.7 weight 3.620 > item osd.8 weight 3.620 > item osd.9 weight 3.620 > } > osdgroup b02s08-osdgroupC { > id -83 # do not change unnecessarily > # weight 19.920 > alg straw > hash 0 # rjenkins1 > item osd.10 weight 3.620 > item osd.11 weight 3.620 > item osd.12 weight 3.620 > item osd.13 weight 3.620 > item osd.14 weight 5.440 > } > osdgroup b02s08-osdgroupD { > id -84 # do not change unnecessarily > # weight 19.920 > alg straw > hash 0 # rjenkins1 > item osd.15 weight 3.620 > item osd.16 weight 3.620 > item osd.17 weight 3.620 > item osd.18 weight 3.620 > item osd.19 weight 5.440 > } > host b02s08 { > id -80 # do not change unnecessarily > # weight 76.040 > alg straw > hash 0 # rjenkins1 > item b02s08-osdgroupA weight 18.100 > item b02s08-osdgroupB weight 18.100 > item b02s08-osdgroupC weight 19.920 > item b02s08-osdgroupD weight 19.920 > } > osdgroup b02s12-osdgroupA { > id -121 # do not change unnecessarily > # weight 18.100 > alg straw > hash 0 # rjenkins1 > item osd.20 weight 3.620 > item osd.21 weight 3.620 > item osd.22 weight 3.620 > item osd.23 weight 3.620 > item osd.24 weight 3.620 > } > osdgroup b02s12-osdgroupB { > id -122 # do not change unnecessarily > # weight 18.100 > alg straw > hash 0 # rjenkins1 > item osd.25 weight 3.620 > item osd.26 weight 3.620 > item osd.27 weight 3.620 > item osd.28 weight 3.620 > item osd.29 weight 3.620 > } > osdgroup b02s12-osdgroupC { > id -123 # do not change unnecessarily > # weight 19.920 > alg straw > hash 0 # rjenkins1 > item osd.30 weight 3.620 > item osd.31 weight 3.620 > item osd.32 weight 3.620 > item osd.33 weight 3.620 > item osd.34 weight 5.440 > } > osdgroup b02s12-osdgroupD { > id -124 # do not change unnecessarily > # weight 19.920 > alg straw > hash 0 # rjenkins1 > item osd.35 weight 3.620 > item osd.36 weight 3.620 > item osd.37 weight 3.620 > item osd.38 weight 3.620 > item osd.39 weight 5.440 > } > host b02s12 { > id -120 # do not change unnecessarily > # weight 76.040 > alg straw > hash 0 # rjenkins1 > item b02s12-osdgroupA weight 18.100 > item b02s12-osdgroupB weight 18.100 > item b02s12-osdgroupC weight 19.920 > item b02s12-osdgroupD weight 19.920 > } > root replicated-T1 { > id -1 # do not change unnecessarily > # weight 152.080 > alg straw > hash 0 # rjenkins1 > item b02s08 weight 76.040 > item b02s12 weight 76.040 > } > rack b02 { > id -20 # do not change unnecessarily > # weight 152.080 > alg straw > hash 0 # rjenkins1 > item b02s08 weight 76.040 > item b02s12 weight 76.040 > } > site erbus { > id -10 # do not change unnecessarily > # weight 152.080 > alg straw > hash 0 # rjenkins1 > item b02 weight 152.080 > } > > # rules > rule replicated { > ruleset 0 > type replicated > min_size 1 > max_size 10 > step take replicated-T1 > step choose firstn 0 type host > step chooseleaf firstn 0 type osdgroup > step emit > } > > # end crush map > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
