[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-06 Thread Frank Schilder
Cc: Alexander E. Patrakov; Mario Giammarco; ceph-users Subject: Re: [ceph-users] Re: Worst thing that can happen if I have size= 2 How do you achieve that? 2 hours? Install new drive for db is 10min of DC engineer hand work (if drive is HHHL and need power off server). Then after server is boots your

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-06 Thread Frank Schilder
egards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Mario Giammarco Sent: 05 February 2021 21:10:33 To: Eneko Lacunza Cc: Ceph Users Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 Il giorno gio 4 feb 202

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-05 Thread Mark Lehrer
avalon.org.ua > Cc: "ceph-users" > Sent: Wednesday, February 3, 2021 4:43:08 AM > Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 > > On Wed, 2021-02-03 at 09:39 +, Max Krasilnikov wrote: > > > if a OSD becomes unavailble (broken disk, rebo

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-05 Thread Simon Ironside
On 05/02/2021 20:10, Mario Giammarco wrote: It is not that a morning I wake up and put some random hardware together, I followed guidelines. The result should be: - if a disk (or more) brokes work goes on - if a server brokes the VMs on the server start on another server and work goes on. The

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-05 Thread Mario Giammarco
Il giorno gio 4 feb 2021 alle ore 12:19 Eneko Lacunza ha scritto: > Hi all, > > El 4/2/21 a las 11:56, Frank Schilder escribió: > >> - three servers > >> - three monitors > >> - 6 osd (two per server) > >> - size=3 and min_size=2 > > This is a set-up that I would not run at all. The first one

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-05 Thread Frank Schilder
alert at night, I want to be able to turn around and continue sleeping. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Alexander E. Patrakov Sent: 04 February 2021 11:35:27 To: Mario Giammarco Cc: ceph-users Subje

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-05 Thread Dan van der Ster
volume lvm create --osd-id 1234 Cheers, Dan > > thanks a lot, > > Samuel > > > > > huxia...@horebdata.cn > > > From: Dan van der Ster > Date: 2021-02-04 11:57 > To: Mario Giammarco > CC: Ceph Users > Subject: [ceph-u

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread huxia...@horebdata.cn
From: Dan van der Ster Date: 2021-02-04 11:57 To: Mario Giammarco CC: Ceph Users Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 On Thu, Feb 4, 2021 at 11:30 AM Mario Giammarco wrote: > > > > Il giorno mer 3 feb 2021 alle ore 21:22 Dan van der Ster > h

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Anthony D'Atri
> > Maybe the weakest thing in that configuration is having 2 OSDs per node; osd > nearfull must be tuned accordingly so that no OSD goes beyond about 0.45, so > that in case of failure of one disk, the other OSD in the node has enough > space for healing replication. > A careful setting

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Eneko Lacunza
Hi all, El 4/2/21 a las 11:56, Frank Schilder escribió: - three servers - three monitors - 6 osd (two per server) - size=3 and min_size=2 This is a set-up that I would not run at all. The first one is, that ceph lives on the law of large numbers and 6 is a small number. Hence, your OSD

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Frank Schilder
n van der Ster Sent: 04 February 2021 11:57:38 To: Mario Giammarco Cc: Ceph Users Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 On Thu, Feb 4, 2021 at 11:30 AM Mario Giammarco wrote: > > > > Il giorno mer 3 feb 2021 alle ore 21:22 Dan van der Ster > ha

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Frank Schilder
== Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Mario Giammarco Sent: 04 February 2021 11:29:49 To: Dan van der Ster Cc: Ceph Users Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 Il giorno mer 3 feb 2021 alle ore 21:22

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Robert Sander
Hi, Am 04.02.21 um 12:10 schrieb Frank Schilder: > Going to 2+2 EC will not really help On such a small cluster you cannot even use EC because there are not enough independent hosts. As a rule of thumb there should be k+m+1 hosts in a cluster AFAIK. Regards -- Robert Sander Heinlein Support

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Dan van der Ster
On Thu, Feb 4, 2021 at 11:30 AM Mario Giammarco wrote: > > > > Il giorno mer 3 feb 2021 alle ore 21:22 Dan van der Ster > ha scritto: >> >> >> Lastly, if you can't afford 3x replicas, then use 2+2 erasure coding if >> possible. >> > > I will investigate I heard that erasure coding is slow. > >

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Alexander E. Patrakov
There is a big difference between traditional RAID1 and Ceph. Namely, with Ceph, there are nodes where OSDs are running, and these nodes need maintenance. You want to be able to perform maintenance even if you have one broken OSD, that's why the recommendation is to have three copies with Ceph.

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Mario Giammarco
Il giorno mer 3 feb 2021 alle ore 21:22 Dan van der Ster ha scritto: > > Lastly, if you can't afford 3x replicas, then use 2+2 erasure coding if > possible. > > I will investigate I heard that erasure coding is slow. Anyway I will write here the reason of this thread: In my customers I have

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Mario Giammarco
Il giorno gio 4 feb 2021 alle ore 00:33 Simon Ironside < sirons...@caffetine.org> ha scritto: > > > On 03/02/2021 19:48, Mario Giammarco wrote: > > To labour Dan's point a bit further, maybe a RAID5/6 analogy is better > than RAID1. Yes, I know we're not talking erasure coding pools here but >

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-04 Thread Mario Giammarco
Hi Federico, here I am not mixing raid1 with ceph. I am doing a comparison: is it safer to have a server with raid1 disks or two servers with ceph and size=2 min_size=1 ? We are talking about real world examples where a customer is buying a new server and want to choose. Il giorno gio 4 feb 2021

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Simon Ironside
On 03/02/2021 19:48, Mario Giammarco wrote: It is obvious and a bit paranoid because many servers on many customers run on raid1 and so you are saying: yeah you have two copies of the data but you can broke both. Consider that in ceph recovery is automatic, with raid1 some one must manually

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Dan van der Ster
Ceph is multiple factors more risky with min_size 1 than good old raid1: With raid1, having disks A and B, when disk A fails, you start recovery to a new disk A'. If disk B fails during recovery then you have a disaster. With Ceph, we have multiple servers and multiple disks: When an OSD fails

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Mario Giammarco
Thanks Simon and thanks to other people that have replied. Sorry but I try to explain myself better. It is evident to me that if I have two copies of data, one brokes and while ceph creates again a new copy of the data also the disk with the second copy brokes you lose the data. It is obvious and

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Simon Ironside
On 03/02/2021 09:24, Mario Giammarco wrote: Hello, Imagine this situation: - 3 servers with ceph - a pool with size 2 min 1 I know perfectly the size 3 and min 2 is better. I would like to know what is the worst thing that can happen: Hi Mario, This thread is worth a read, it's an oldie but

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Adam Boyhan
ednesday, February 3, 2021 8:50 AM To: Magnus HAGDORN Cc: ceph-users Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 Isn't this somewhat reliant on the OSD type? Redhat/Micron/Samsung/Supermicro have all put out white papers backing the idea of 2 copies on NVMe's as

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Martin Verges
NVMe's as safe for production. > > > From: "Magnus HAGDORN" > To: pse...@avalon.org.ua > Cc: "ceph-users" > Sent: Wednesday, February 3, 2021 4:43:08 AM > Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 > > On Wed, 2021-02-03 at 09:39 +

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread DHilsbos
Subject: [ceph-users] Re: Worst thing that can happen if I have size= 2 Isn't this somewhat reliant on the OSD type? Redhat/Micron/Samsung/Supermicro have all put out white papers backing the idea of 2 copies on NVMe's as safe for production. From: "Magnus HAGDORN" To: pse...@ava

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Magnus HAGDORN
On Wed, 2021-02-03 at 09:39 +, Max Krasilnikov wrote: > > if a OSD becomes unavailble (broken disk, rebooting server) then > > all > > I/O to the PGs stored on that OSD will block until replication > > level of > > 2 is reached again. So, for a highly available cluster you need a > >

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Max Krasilnikov
День добрий! Wed, Feb 03, 2021 at 09:29:52AM +, Magnus.Hagdorn wrote: > if a OSD becomes unavailble (broken disk, rebooting server) then all > I/O to the PGs stored on that OSD will block until replication level of > 2 is reached again. So, for a highly available cluster you need a >

[ceph-users] Re: Worst thing that can happen if I have size= 2

2021-02-03 Thread Magnus HAGDORN
if a OSD becomes unavailble (broken disk, rebooting server) then all I/O to the PGs stored on that OSD will block until replication level of 2 is reached again. So, for a highly available cluster you need a replication level of 3 On Wed, 2021-02-03 at 10:24 +0100, Mario Giammarco wrote: > Hello,