Well, what's "best" really depends on your needs and use-case. The general advise which has been floated several times now is to have at least N+2 entities of your failure domain in your cluster. So for example if you run with size=3 then you should have at least 5 OSDs if your failure domain is OSD and 5 hosts if your failure domain is host.
On Sat, Jun 3, 2017 at 3:20 AM, Oleg Obleukhov <leoleov...@gmail.com> wrote: > But what would be the best? Have 3 servers and how many osd? > Thanks! > > On 2 Jun 2017, at 17:09, David Turner <drakonst...@gmail.com> wrote: > > That's good for testing in the small scale. For production I would > revisit using size 3. Glad you got it working. > > On Fri, Jun 2, 2017 at 11:02 AM Oleg Obleukhov <leoleov...@gmail.com> > wrote: > >> Thanks to everyone, >> problem is solved by: >> ceph osd pool set cephfs_metadata size 2 >> ceph osd pool set cephfs_data size 2 >> >> Best, Oleg. >> >> On 2 Jun 2017, at 16:15, Oleg Obleukhov <leoleov...@gmail.com> wrote: >> >> Hello, >> I am playing around with ceph (ceph version 10.2.7 ( >> 50e863e0f4bc8f4b9e31156de690d765af245185)) on Debian Jessie and I build >> a test setup: >> >> $ ceph osd tree >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> -1 0.01497 root default >> -2 0.00499 host af-staging-ceph01 >> 0 0.00499 osd.0 up 1.00000 1.00000 >> -3 0.00499 host af-staging-ceph02 >> 1 0.00499 osd.1 up 1.00000 1.00000 >> -4 0.00499 host af-staging-ceph03 >> 2 0.00499 osd.2 up 1.00000 1.00000 >> >> So I have 3 osd on 3 servers. >> I also created 2 pools: >> >> ceph osd dump | grep 'replicated size' >> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_ruleset 0 >> object_hash rjenkins pg_num 32 pgp_num 32 last_change 33 flags hashpspool >> crash_replay_interval 45 stripe_width 0 >> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_ruleset 0 >> object_hash rjenkins pg_num 32 pgp_num 32 last_change 31 flags hashpspool >> stripe_width 0 >> >> Now I am testing failover and kill one of servers: >> ceph osd tree >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> -1 0.01497 root default >> -2 0.00499 host af-staging-ceph01 >> 0 0.00499 osd.0 up 1.00000 1.00000 >> -3 0.00499 host af-staging-ceph02 >> 1 0.00499 osd.1 down 1.00000 1.00000 >> -4 0.00499 host af-staging-ceph03 >> 2 0.00499 osd.2 up 1.00000 1.00000 >> >> And now it stuck in the recovery state: >> ceph -s >> cluster 6b5ff07a-7232-4840-b486-6b7906248de7 >> health HEALTH_WARN >> 64 pgs degraded >> 18 pgs stuck unclean >> 64 pgs undersized >> recovery 21/63 objects degraded (33.333%) >> 1/3 in osds are down >> 1 mons down, quorum 0,2 af-staging-ceph01,af-staging-ceph03 >> monmap e1: 3 mons at {af-staging-ceph01=10.36.0. >> 121:6789/0,af-staging-ceph02=10.36.0.122:6789/0,af-staging- >> ceph03=10.36.0.123:6789/0} >> election epoch 38, quorum 0,2 af-staging-ceph01,af-staging- >> ceph03 >> fsmap e29: 1/1/1 up {0=af-staging-ceph03.crm.ig.local=up:active}, >> 2 up:standby >> osdmap e78: 3 osds: 2 up, 3 in; 64 remapped pgs >> flags sortbitwise,require_jewel_osds >> pgmap v334: 64 pgs, 2 pools, 47129 bytes data, 21 objects >> 122 MB used, 15204 MB / 15326 MB avail >> 21/63 objects degraded (33.333%) >> 64 active+undersized+degraded >> >> And if I kill one more node I lose access to mounted file system on >> client. >> Normally I would expect replica-factor to be respected and ceph should >> create the missing copies of degraded pg. >> >> I was trying to rebuild the crush map and it looks like this, but this >> did not help: >> rule replicated_ruleset { >> ruleset 0 >> type replicated >> min_size 1 >> max_size 10 >> step take default >> step chooseleaf firstn 0 type osd >> step emit >> } >> >> # end crush map >> >> Would very appreciate help, >> Thank you very much in advance, >> Oleg. >> >> >> > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com