Re: installing Debian 10 to 3 hdds as one big system

[tomas]

On Sun, Aug 11, 2019 at 04:44:32PM -0700, David Christensen wrote:

[...]

> So, "Linux MD RAID 10" can be RAID 1, RAID 10, or something totally
> different.
> 
> 
> Co-opting and redefining standard terms is bad engineering.

To be honest, and as I remember, the term RAID-10 was doomed from the
get-go. Each manufacturer had a slightly different take on it, so at
the time I filed it under "whatever the marketing department du jour
just came up with". Kind of like the mauve database [1]. This was long
before Linux started doing RAID.

Cheers
[1] https://dilbert.com/strip/1995-11-17

-- t


signature.asc
Description: Digital signature

Re: installing Debian 10 to 3 hdds as one big system

[David Christensen]

On 8/11/19 12:42 AM, Reco wrote:

On Sat, Aug 10, 2019 at 05:25:30PM -0700, David Christensen wrote:

On 8/10/19 3:55 PM, Pascal Hambourg wrote:

Le 10/08/2019 à 19:27, David Christensen a écrit :

On 8/10/19 4:35 AM, Andy Smith wrote:

Personally I would use the three devices as a RAID-10 which would
result in half the capacity of the total (768G) and you could
withstand the loss of any one device.


RAID 10 requires 4 drives:


Not Linux implemention of RAID 10, which requires at least 2 drives.


Do you have a URL or man page that describes this?



...

https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Raid_10



Thanks for the pointer.  STFW:

https://en.wikipedia.org/wiki/RAID_10

"... a nonstandard definition of "RAID 10" was created for the Linux
MD driver ..."

https://en.wikipedia.org/wiki/RAID#Non-standard_levels

https://en.wikipedia.org/wiki/Linux_MD_RAID_10#Linux_MD_RAID_10


So, "Linux MD RAID 10" can be RAID 1, RAID 10, or something totally 
different.



Co-opting and redefining standard terms is bad engineering.


David

Re: installing Debian 10 to 3 hdds as one big system

[Andy Smith]

Hello,

On Sat, Aug 10, 2019 at 11:18:49AM -0700, David Christensen wrote:
> But, as others have said, two HDD's and one SDD in JBOD (or RAID)
> is not optimal.

On the other hand it's probably not going to be worse than 3 HDDs.
All the ways of trying to exploit the speed of the SSD are quite
complicated so if 1x SSD and 2x HDD is what you have to work with,
just putting them all together in a RAID isn't necessarily a bad
idea.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: installing Debian 10 to 3 hdds as one big system

[Pascal Hambourg]

Le 11/08/2019 à 09:42, Reco a écrit :


The big question is - why would anyone make a RAID10 consisting of two
drives. It's impossible to reshape it (mdadm does not support it for
RAID10), it's I/O characteristics are indistinguishable from RAID1.


With two drives the default RAID 10 "near" layout is identical to RAID 
1. But the RAID 10 "far" layout is expected to provide better sequential 
read speed than RAID 1 (like RAID 0).




"RAID-10 has a layout ("far") which can provide sequential read 
throughput that scales by number of drives, rather than number of RAID-1 
pairs. You can get about 95 % of the performance of the RAID-0 with same 
amount of drives."

Re: installing Debian 10 to 3 hdds as one big system

[Reco]

Hi.

On Sat, Aug 10, 2019 at 05:25:30PM -0700, David Christensen wrote:
> On 8/10/19 3:55 PM, Pascal Hambourg wrote:
> > Le 10/08/2019 à 19:27, David Christensen a écrit :
> > > On 8/10/19 4:35 AM, Andy Smith wrote:
> > > > Personally I would use the three devices as a RAID-10 which would
> > > > result in half the capacity of the total (768G) and you could
> > > > withstand the loss of any one device.
> > > 
> > > RAID 10 requires 4 drives:
> > 
> > Not Linux implemention of RAID 10, which requires at least 2 drives.
> 
> Do you have a URL or man page that describes this?

[1] says (emphasis mine):

Raid 10 is a special linux mode which stores multiple copies of the data
across several drives. There must be at least as many drives as there
are copies - if the *two are equal* it is equivalent to raid-1.

...

Conversion between raid-0 and raid-10 is supported - to convert to any
other raid you will have to go via raid-0 - backup, BACKUP, BACKUP!!!
The code does not appear to support conversion between raid-1 and
raid-10 which should be easy for a *two-device* array.


The big question is - why would anyone make a RAID10 consisting of two
drives. It's impossible to reshape it (mdadm does not support it for
RAID10), it's I/O characteristics are indistinguishable from RAID1.

Reco

https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Raid_10

Re: installing Debian 10 to 3 hdds as one big system

[David Christensen]

On 8/10/19 3:55 PM, Pascal Hambourg wrote:

Le 10/08/2019 à 19:27, David Christensen a écrit :

On 8/10/19 4:35 AM, Andy Smith wrote:

Personally I would use the three devices as a RAID-10 which would
result in half the capacity of the total (768G) and you could
withstand the loss of any one device.


RAID 10 requires 4 drives:


Not Linux implemention of RAID 10, which requires at least 2 drives.


Do you have a URL or man page that describes this?


David

Re: installing Debian 10 to 3 hdds as one big system

[Pascal Hambourg]

Le 10/08/2019 à 19:27, David Christensen a écrit :

On 8/10/19 4:35 AM, Andy Smith wrote:

Personally I would use the three devices as a RAID-10 which would
result in half the capacity of the total (768G) and you could
withstand the loss of any one device.


RAID 10 requires 4 drives:


Not Linux implemention of RAID 10, which requires at least 2 drives.

Re: installing Debian 10 to 3 hdds as one big system

[deloptes]

Pavel Vlček wrote:

> I have computer with 3 hdds. One is ssd, 2 others are hdd. I want to
> install Debian 10 to all 3 disks as one big system. What to use, raid or
> lvm? I found an issue with lvm, when I want to create lvm, it shows you
> can use $minsize and $maxsize, but all disks are 512, Gb. I want to
> start with the ssd, it is sdc device. I know, how to create the lvm with
> textual installer, but I have problem expanding it to next two hdds,
> /dev/sda and sdb, so I am doing something wrong. I want to use all data
> in one lvm partition, so / for everything.
> Can you help please?

May be it will help to explain what is the desired usage of the system as
many mentioned SDD and HDD is not good to mix. In the case I would use the
SDD for the operating system and not sensitive data that is written more
often for example a scratch box.
The HDDs you can put in RAID1 and probably now and then backup your
configuration from the SDD, so that you can recover if the SDD fails. It is
not likely but it happens more often than you can think.

Re: installing Debian 10 to 3 hdds as one big system

[David Christensen]

On 8/10/19 4:03 AM, Pavel Vlček wrote:

Hi all,

because I was not registered when I sent this message here at the first 
time, I am sending it again.


I have computer with 3 hdds. One is ssd, 2 others are hdd. I want to 
install Debian 10 to all 3 disks as one big system. What to use, raid or 
lvm? I found an issue with lvm, when I want to create lvm, it shows you 
can use $minsize and $maxsize, but all disks are 512, Gb. I want to 
start with the ssd, it is sdc device. I know, how to create the lvm with 
textual installer, but I have problem expanding it to next two hdds, 
/dev/sda and sdb, so I am doing something wrong. I want to use all data 
in one lvm partition, so / for everything.

Can you help please?
Thanks,
Pavel


You are describing JBOD:

https://en.wikipedia.org/wiki/JBOD


I have done JBOD on LVM.  See lvm(8).


I have done RAID1 on mdadm; I believe mdadm can do JBOD,  See mdadm(8).


If this is a new install, you should be able to achieve either or both 
via the Debian Installer by selecting manual partitioning.



But, as others have said, two HDD's and one SDD in JBOD (or RAID) is not 
optimal.  Without specifics such as anticipated usage, data storage 
requirements, drive makes and models, available drive bays, HBA/ 
motherboard makes, models, and available ports, etc.,  it is difficult 
to make recommendations.  In general, I put boot, swap, and root on one 
small SSD and put bulk data on RAID large HDD's.



David

Re: installing Debian 10 to 3 hdds as one big system

[David Christensen]

On 8/10/19 4:35 AM, Andy Smith wrote:

Personally I would use the three devices as a RAID-10 which would
result in half the capacity of the total (768G) and you could
withstand the loss of any one device.


RAID 10 requires 4 drives:

https://en.wikipedia.org/wiki/Raid10#RAID_10_(RAID_1+0)

RAID 10, also called RAID 1+0 and sometimes RAID 1&0, is similar to RAID 
01 with an exception that two used standard RAID levels are layered in 
the opposite order; thus, RAID 10 is a stripe of mirrors.



David

Re: installing Debian 10 to 3 hdds as one big system

[Pascal Hambourg]

Le 10/08/2019 à 13:03, Pavel Vlček a écrit :


I have computer with 3 hdds. One is ssd, 2 others are hdd. I want to 
install Debian 10 to all 3 disks as one big system.


What do you mean by "one big system" ? One big filesystem ? Why ?


What to use, raid or lvm?


If you are concerned with performance, do not mix SSD and HDD in an LVM 
group or a RAID array (except RAID 1 with --write-mostly as already 
mentionned).


If you are concerned with disk fault tolerance, use RAID with redundancy 
(not RAID "linear" or RAID 0).


Note that you can set up LVM on top of RAID.

Re: installing Debian 10 to 3 hdds as one big system

[Andy Smith]

Hi Pavel,

On Sat, Aug 10, 2019 at 01:03:10PM +0200, Pavel Vlček wrote:
> I have computer with 3 hdds. One is ssd, 2 others are hdd. I want to install
> Debian 10 to all 3 disks as one big system. What to use, raid or lvm?

Personally I would use the three devices as a RAID-10 which would
result in half the capacity of the total (768G) and you could
withstand the loss of any one device.

You could instead do RAID-5 but I do not like parity RAIDs. In this
case that would give you 2 devices worth of capacity and again any one
device could fail. It won't perform as well as RAID-10.

Other options that include redundancy would be btrfs or zfs.

I would not do the redundancy in LVM.

If not using btrfs or zfs, I would use LVM afterwards on the RAID
device for management purposes.

You can do all of this (except zfs) in the Debian installer.

There is an MD feature called "write-mostly" which you can set on
devices to tell the kernel that no reads should go to these devices
unless absolutely necessary. The usual use of this is in mixed
rotational and flash setups to try to encourage reads to come from
the much faster flash devices. This could be of real benefit to you
but sadly it doesn't work with anything except RAID-1.

Other interesting approaches could be:

- RAID-1 of the rotational devices then use the SSD in lvmcache. In
  writethrough mode it is safe to lose the (non-redundant) cache
  device:

https://manpages.debian.org/buster/lvm2/lvmcache.7.en.html

- RAID-1 of the rotational devices then bcache on the SSD:

https://bcache.evilpiepirate.org/

Personally I don't find bcache mature enough and while lvmcache I
did find to be safe, I didn't find that it improved performance that
much, probably not enough to dedicate a third of my total capacity
to it.

If performance was my overriding concern I might actually do a 3-way
RAID-1 with the two HDDs set to write-mostly. Only 512G capacity,
can lose any two devices, good read performance.

> I know, how to create the lvm with textual installer,
> but I have problem expanding it to next two hdds, /dev/sda and sdb

I would not do this, but…

# Mark the new device as an LVM PV
# pvcreate /dev/sda

# Extend your current volume group to use the new PV
# vgextend /dev/your_vg_name /dev/sda

At this point you have added the capacity of /dev/sda to the
existing volume group, but all your existing logical volumes still
reside on the original PV alone. You can now convert them to have
their extents mirrored:

# lvconvert -m1 /dev/your_vg_name/some_lv

The mirrored extents will be on /dev/sda because that is the only
PV with free extents. If you already added /dev/sdb then the extents
could be mirrored there instead. If you want to specify where the
mirrored extents should go then you can do so by appending the PV
device path to the above lvconvert command.

You can instead/also stripe, using the --stripes option to lvconvert
(or lvcreate, for new LVs). In this setup there would be no
redundancy though, which is too bad for me to consider.

Not doing anything special would leave your LVs being allocated
sequentially from whichever PV has capacity, in this setup resulting
in no redundancy and max performance of one device, so that would be
the worst setup.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: installing Debian 10 to 3 hdds as one big system

[Rene Engelhard]

Hi,

On Sat, Aug 10, 2019 at 01:03:10PM +0200, Pavel Vlček wrote:
> I have computer with 3 hdds. One is ssd, 2 others are hdd. I want to install
> Debian 10 to all 3 disks as one big system. What to use, raid or lvm? I

Depends. Do you want to have 1.5 Gigs combined? (LVM)

[...]
> doing something wrong. I want to use all data in one lvm partition, so / for
> everything.

So something or the systen files is/are potentially split between ssd
and hdd and you don't know where to get it from? Sounds like a "no idea"
performance-wise.

If you had 3 SSDs or 3 HDDs, ok, but..

If you want that "big space" and not care about data security if one
fsailed I'd do system on a partition on the SSD and then do LVM on the rest of
the SSD+the HDDs. Then you at least have the system on the SSD completely.

(And you know LVM does nothing for security if one of your disk fails?).

If you can live with "just" 512 GB home I'd say: system on the SSD
(although some "waste" of diskspace...),
RAID1 for the two HDDs mounted on /home for data security if one disk
fails (of course doesn't save a backup for rm actions or so..)

Regards,

Rene