[ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe
Hi!
I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to
CephFS.
This storage is used for backup images (large sequential reads and writes).
To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
following profile:
ceph osd erasure-code-profile set myprofile \
k=3 \
m=2 \
ruleset-failure-domain=rack
Performance is not the first priority, this is why I do not plan to
outsource WAL/DB (broken NVMe = broken OSDs is more administrative overhead
then single OSDs).
Disks are attached by SAS multipath, throughput in general is no problem
but I did not test with ceph yet.
Is anyone using CephFS + bluestore + ec 3/2 + without WAL/DB-dev and is it
working well?
Thank you.
Kevin
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe
Not 3+2, but we run 4+2, 6+2, 6+3, 5+3, and 8+3 with cephfs in
production. Most of them are HDDs without separate DB devices.
Paul
2018-08-22 14:27 GMT+02:00 Kevin Olbrich :
> Hi!
>
> I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to
> CephFS.
> This storage is used for backup images (large sequential reads and writes).
>
> To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
> following profile:
>
> ceph osd erasure-code-profile set myprofile \
>k=3 \
>m=2 \
>ruleset-failure-domain=rack
>
> Performance is not the first priority, this is why I do not plan to
> outsource WAL/DB (broken NVMe = broken OSDs is more administrative overhead
> then single OSDs).
> Disks are attached by SAS multipath, throughput in general is no problem but
> I did not test with ceph yet.
>
> Is anyone using CephFS + bluestore + ec 3/2 + without WAL/DB-dev and is it
> working well?
>
> Thank you.
>
> Kevin
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe
I would suggest having some flash media for their own OSDs to put the
cephfs metadata pool onto. That was a pretty significant boost for me when
I moved the metadata pool onto flash media. My home setup is only 3 nodes
and is running EC 2+1 on pure HDD OSDs with metadata on SSDs. It's been
running stable and fine for a couple years now. I wouldn't suggest running
EC 2+1 for any data you can't lose, but I can replace anything in there
with some time.
On Wed, Aug 22, 2018 at 8:43 AM Paul Emmerich
wrote:
> Not 3+2, but we run 4+2, 6+2, 6+3, 5+3, and 8+3 with cephfs in
> production. Most of them are HDDs without separate DB devices.
>
>
>
> Paul
>
> 2018-08-22 14:27 GMT+02:00 Kevin Olbrich :
> > Hi!
> >
> > I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to
> > CephFS.
> > This storage is used for backup images (large sequential reads and
> writes).
> >
> > To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
> > following profile:
> >
> > ceph osd erasure-code-profile set myprofile \
> >k=3 \
> >m=2 \
> >ruleset-failure-domain=rack
> >
> > Performance is not the first priority, this is why I do not plan to
> > outsource WAL/DB (broken NVMe = broken OSDs is more administrative
> overhead
> > then single OSDs).
> > Disks are attached by SAS multipath, throughput in general is no problem
> but
> > I did not test with ceph yet.
> >
> > Is anyone using CephFS + bluestore + ec 3/2 + without WAL/DB-dev and is
> it
> > working well?
> >
> > Thank you.
> >
> > Kevin
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
> Paul Emmerich
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90 <+49%2089%20189658590>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe
I also have 2+1 (still only 3 nodes), and 3 replicated. I also moved the
meta datapool to ssds.
What is nice with the cephfs, you can have folders in your filesystem on
the ec21 pool for not so important data and the rest will be 3x
replicated.
I think the single session performance is not going to give you same
performance as the raid. But you can compensate that by doing your
backup in parallel.
-Original Message-
From: Kevin Olbrich [mailto:k...@sv01.de]
Sent: woensdag 22 augustus 2018 14:28
To: ceph-users
Subject: [ceph-users] HDD-only CephFS cluster with EC and without
SSD/NVMe
Hi!
I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to
CephFS.
This storage is used for backup images (large sequential reads and
writes).
To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
following profile:
ceph osd erasure-code-profile set myprofile \
k=3 \
m=2 \
ruleset-failure-domain=rack
Performance is not the first priority, this is why I do not plan to
outsource WAL/DB (broken NVMe = broken OSDs is more administrative
overhead then single OSDs).
Disks are attached by SAS multipath, throughput in general is no problem
but I did not test with ceph yet.
Is anyone using CephFS + bluestore + ec 3/2 + without WAL/DB-dev and is
it working well?
Thank you.
Kevin
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe
On Wed, Aug 22, 2018 at 1:28 PM Kevin Olbrich wrote:
>
> Hi!
>
> I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to CephFS.
> This storage is used for backup images (large sequential reads and writes).
>
> To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
> following profile:
>
> ceph osd erasure-code-profile set myprofile \
>k=3 \
>m=2 \
>ruleset-failure-domain=rack
>
> Performance is not the first priority, this is why I do not plan to outsource
> WAL/DB (broken NVMe = broken OSDs is more administrative overhead then single
> OSDs).
> Disks are attached by SAS multipath, throughput in general is no problem but
> I did not test with ceph yet.
>
> Is anyone using CephFS + bluestore + ec 3/2 + without WAL/DB-dev and is it
> working well?
I have a very small home cluster that's 6x OSDs over 3 nodes, using EC
on bluestore on spinning disks. I don't have benchmarks, but it was
usable for a few TB of backups.
John
>
> Thank you.
>
> Kevin
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
