Re: [Gluster-users] To RAID or not to RAID...

[Hu Bert]

Hi,

our old setup is not really comparable, but i thought i'd drop some
lines... we once had a Distributed-Replicate setup with 4 x 3 = 12
disks (10 TB hdd). Simple JBOD, every disk == brick. Was running
pretty good, until one of the disks died. The restore (reset-brick)
took about a month, because the application has a quite high I/O and
therefore slows down the volume and the disks.

Next step: take servers with 10x10TB disks and build a RAID 10; raid
array == brick, replicate volume (1 x 3 = 3). When a disk fails, you
only have to rebuild the SW RAID which takes about 3-4 days, plus the
periodic redundany checks. This was way better than the
JBOD/reset-scenario before. But still not optimal. Upcoming step:
build a distribute-replicate with lots of SSDs (maybe again with a
RAID underneath) .

tl;dr what i wanted to say: we waste a lot of disks. It simply depends
on which setup you have and how to handle the situation when one of
the disks fails - and it will! ;-(


regards
Hubert

Am Di., 14. Jan. 2020 um 12:36 Uhr schrieb Markus Kern :
>
>
> Greetings again!
>
> After reading RedHat documentation regarding optimizing Gluster storage
> another question comes to my mind:
>
> Let's presume that I want to go the distributed dispersed volume way.
> Three nodes which two bricks each.
> According to RedHat's recommendation, I should use RAID6 as underlying
> RAID for my planned workload.
> I am frightened by that "waste" of disks in such a case:
> When each brick is a RAID6, I would "loose" two disks per brick - 12
> lossed disks in total.
> In addition to this, distributed dispersed volume adds another layer of
> lossed disk space.
>
> Am I wrong here? Maybe I didn't understand the recommendations wrong?
>
> Markus
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] To RAID or not to RAID...

[Strahil]

Hi Markus,


You are right  .I think that the 3 node setup matches distributed volume.

According to 
https://docs.gluster.org/en/latest/Administrator%20Guide/Setting%20Up%20Volumes/

Dispersed  volumes  use erasure codes to have a 'parity' on a separate brick. 
In such case you can afford to  loose a brick without loosing data and you will 
need more bricks.

Yet, I don't see anything about RAID6 being required.

Use the gluster's official documentation (if possible) as it is the most recent 
info.

Maybe you can share the ammount of disks , raid controllers and servers you 
have and your tolerance to data loss. Then I can share my thoughts on the 
possible volume types.

Best Regards,
Strahil Nikolov

On Jan 14, 2020 20:33, Markus Kern  wrote:
>
> Hi Strahil, 
>
> thanks for you answer - but now I am completely lost :) 
>
> From this documentation: 
> https://docs.oracle.com/cd/E52668_01/F10040/html/gluster-312-volume-distr-disp.html
>  
>
> "As a dispersed volume must have a minimum of three bricks, a 
> distributed dispersed volume must have at least six bricks. For example, 
> six nodes with one brick, or three nodes with two bricks on each node 
> are needed for this volume type." 
>
> So for a distributed dispersed volume I need at least six bricks. If 
> each brick is a RAID6, I have 6 x 2 Parity disks = 12 disks for parity. 
>
> In your example you only have one brick per node in a three node setup. 
> This is no distributed dispersed volume then, right? 
>
> A confused Markus 
>
>
> Am 14.01.2020 16:29, schrieb Strahil: 
> > Hi Markus, 
> > 
> > Distributed dispersed volume is just LVM's linear LV -> so in case of 
> > brick failiure - you loose the data on it. 
> > 
> > Raid 6 requires  2 disks  for parity, so you can make a large RAID6 
> > and use that as a single brick - so the disks that hold  the parity 
> > data are  only 6 ( 3 nodes x 2 disks). 
> > 
> > Of course  if you have too many disks  for a single raid controller 
> > ,that you can consider a  replica volume  with an arbiter. 
> > 
> > 
> > Best Regards, 
> > Strahil Nikolov 
> > 
> > On Jan 14, 2020 13:36, Markus Kern  wrote: 
> >> 
> >> 
> >> Greetings again! 
> >> 
> >> After reading RedHat documentation regarding optimizing Gluster 
> >> storage 
> >> another question comes to my mind: 
> >> 
> >> Let's presume that I want to go the distributed dispersed volume way. 
> >> Three nodes which two bricks each. 
> >> According to RedHat's recommendation, I should use RAID6 as underlying 
> >> RAID for my planned workload. 
> >> I am frightened by that "waste" of disks in such a case: 
> >> When each brick is a RAID6, I would "loose" two disks per brick - 12 
> >> lossed disks in total. 
> >> In addition to this, distributed dispersed volume adds another layer 
> >> of 
> >> lossed disk space. 
> >> 
> >> Am I wrong here? Maybe I didn't understand the recommendations wrong? 
> >> 
> >> Markus 
> >>  
> >> 
> >> Community Meeting Calendar: 
> >> 
> >> APAC Schedule - 
> >> Every 2nd and 4th Tuesday at 11:30 AM IST 
> >> Bridge: https://bluejeans.com/441850968 
> >> 
> >> NA/EMEA Schedule - 
> >> Every 1st and 3rd Tuesday at 01:00 PM EDT 
> >> Bridge: https://bluejeans.com/441850968 
> >> 
> >> Gluster-users mailing list 
> >> Gluster-users@gluster.org 
> >> https://lists.gluster.org/mailman/listinfo/gluster-users 


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] To RAID or not to RAID...

[Markus Kern]

Hi Strahil,

thanks for you answer - but now I am completely lost :)

From this documentation:
https://docs.oracle.com/cd/E52668_01/F10040/html/gluster-312-volume-distr-disp.html

"As a dispersed volume must have a minimum of three bricks, a 
distributed dispersed volume must have at least six bricks. For example, 
six nodes with one brick, or three nodes with two bricks on each node 
are needed for this volume type."


So for a distributed dispersed volume I need at least six bricks. If 
each brick is a RAID6, I have 6 x 2 Parity disks = 12 disks for parity.


In your example you only have one brick per node in a three node setup. 
This is no distributed dispersed volume then, right?


A confused Markus


Am 14.01.2020 16:29, schrieb Strahil:

Hi Markus,

Distributed dispersed volume is just LVM's linear LV -> so in case of
brick failiure - you loose the data on it.

Raid 6 requires  2 disks  for parity, so you can make a large RAID6
and use that as a single brick - so the disks that hold  the parity
data are  only 6 ( 3 nodes x 2 disks).

Of course  if you have too many disks  for a single raid controller
,that you can consider a  replica volume  with an arbiter.


Best Regards,
Strahil Nikolov

On Jan 14, 2020 13:36, Markus Kern  wrote:



Greetings again!

After reading RedHat documentation regarding optimizing Gluster 
storage

another question comes to my mind:

Let's presume that I want to go the distributed dispersed volume way.
Three nodes which two bricks each.
According to RedHat's recommendation, I should use RAID6 as underlying
RAID for my planned workload.
I am frightened by that "waste" of disks in such a case:
When each brick is a RAID6, I would "loose" two disks per brick - 12
lossed disks in total.
In addition to this, distributed dispersed volume adds another layer 
of

lossed disk space.

Am I wrong here? Maybe I didn't understand the recommendations wrong?

Markus


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Dmitry Melekhov]

04.07.2016 19:01, Matt Robinson пишет:

With mdadm any raid6 (especially with 12 disks) will be rubbish.


Well, this can be offtopic, but could you, please, explain why? (never 
used md raid other than raid1... )


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[tom]

If you go the ZFS route - be absolutely sure you set xattr=sa on all 
filesystems that will hold bricks BEFORE you create bricks on same.  Not doing 
so will cause major problems with data that should be deleted not being 
reclaimed until after a forced dismount or reboot (which can take hours -> days 
if there are several terabytes of data to reclaim.)

Setting it also vastly improves directory and stat() performance.

Setting it after the bricks had been created led to data inconsistencies and 
eventual data loss on a cluster we used to operate.

-t

> On Jul 4, 2016, at 4:35 PM, Lindsay Mathieson  
> wrote:
> 
> On 5/07/2016 12:54 AM, Gandalf Corvotempesta wrote:
>> No suggestions ?
>> 
>> Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" 
>> mailto:gandalf.corvotempe...@gmail.com>> 
>> ha scritto:
>> Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
>> This cluster would be expanded to a maximum of 15 servers in near future.
>> 
>> What do you suggest, a JBOD or a RAID? Which RAID level?
> 
> 
> I setup my much smaller cluster with ZFS RAID10 on each node. 
> - Greatly increased the iops per node
> 
> - auto bitrot detection and repair
> 
> - SSD caches
> 
> - compression clawed back 30% of the disk space I lost to RAID10.
> -- 
> Lindsay Mathieson
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Lindsay Mathieson]

On 5/07/2016 12:54 AM, Gandalf Corvotempesta wrote:


No suggestions ?

Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" 
> ha scritto:


Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
This cluster would be expanded to a maximum of 15 servers in near
future.

What do you suggest, a JBOD or a RAID? Which RAID level?




I setup my much smaller cluster with ZFS RAID10 on each node.

- Greatly increased the iops per node

- auto bitrot detection and repair

- SSD caches

- compression clawed back 30% of the disk space I lost to RAID10.

--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Russell Purinton]

Agreed… It took me almost 2 years of tweaking and testing to get the 
performance I wanted.   

Different workloads require different configurations.Test different 
configurations and find what works best for you!

> On Jul 4, 2016, at 2:15 PM, t...@encoding.com wrote:
> 
> I would highly stress, regardless of whatever solution you choose - make sure 
> you test actual workload performance before going all-in.
> 
> In my testing, performance (esp. iops and latency) decreased as I added 
> bricks and additional nodes.  Since you have many spindles now, I would 
> encourage you to test your workload up to and including the total brick count 
> you ultimately expect.  RAID level and whether it’s md, zfs, or hardware 
> isn’t likely to make as significant of a performance impact as Gluster and 
> its various clients will.  Test failure scenarios and performance 
> characteristics during impairment events thoroughly.  Make sure heals happen 
> as you expect, including final contents of files modified during an 
> impairment.  If you have many small files or directories that will be 
> accessed concurrently, make sure to stress that behavior in your testing.
> 
> Gluster can be great for targeting availability and distribution at low 
> software cost, and I would say as of today at the expense of performance, but 
> as with any scale-out NAS there are limitations and some surprises along the 
> path.
> 
> Good hunting,
> -t
> 
>> On Jul 4, 2016, at 10:44 AM, Gandalf Corvotempesta 
>>  wrote:
>> 
>> 2016-07-04 19:35 GMT+02:00 Russell Purinton :
>>> For 3 servers with 12 disks each, I would do Hardware RAID0 (or madam if 
>>> you don’t have a RAID card) of 3 disks.  So four 3-disk RAID0’s per server.
>> 
>> 3 servers is just to start. We plan to use 5 server in shorter time
>> and up to 15 on production.
>> 
>>> I would set them up as Replica 3 Arbiter 1
>>> 
>>> server1:/brickA server2:/brickC server3:/brickA
>>> server1:/brickB server2:/brickD server3:/brickB
>>> server2:/brickA server3:/brickC server1:/brickA
>>> server2:/brickB server3:/brickD server1:/brickB
>>> server3:/brickA server1:/brickC server2:/brickA
>>> server3:/brickB server1:/brickD server2:/brickB
>>> 
>>> The benefit of this is that you can lose an entire server node (12 disks) 
>>> and all of your data is still accessible.   And you get the same space as 
>>> if they were all in a RAID10.
>>> 
>>> If you lose any disk, the entire 3 disk brick will need to be healed from 
>>> the replica.   I have 20GbE on each server so it doesn’t take long.   It 
>>> copied 20TB in about 18 hours once.
>> 
>> So, any disk failure would me at least 6TB to be recovered via
>> network. This mean an high network utilization and as long gluster
>> doesn't have a dedicated network for replica,
>> this can slow down client access.
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Russell Purinton]

Sorry, example of 5 servers should read

> server1 A & B   replica to server 2 C & D
> server2 A & B   replica to server 3 C & D
> server3 A & B   replica to server 4 C & D
> server4 A & B   replica to server 5 C & D
> server5 A & B   replica to server 1 C & D


Adding each server should be as simple as using the brick-replace command to 
move bricks C and D from server1 onto bricks C and D of the new server.

Then you can add-brick to create 2 new brick replicas from new server A and B 
to server1 C and D.


> On Jul 4, 2016, at 1:54 PM, Russell Purinton  
> wrote:
> 
> The fault tolerance is provided by Gluster replica translator.
> 
> RAID0 to me is preferable to JBOD because you get 3x read performance and 3x 
> write performance.   If performance is not a concern, or if you only have 
> 1GbE, then it may not matter, and you could just do JBOD with a ton of bricks.
> 
> The same method scales to how ever many servers you need… imagine them in a 
> ring…
> 
> server1 A & B   replica to server 2 C & D
> server2 A & B   replica to server 3 C & D
> server3 A & B   replica to server 1 C & D
> 
> Adding a 4th server?  No problem… you can move the reconfigure the bricks to 
> do
> server1 A & B   replica to server 2 C & D
> server2 A & B   replica to server 3 C & D
> server3 A & B   replica to server 4 C & D
> server4 A & B   replica to server 1 C & D
> 
> or 5 servers
> server1 A & B   replica to server 2 C & D
> server2 A & B   replica to server 3 C & D
> server3 A & B   replica to server 4 C & D
> server4 A & B   replica to server 5 C & D
> server5 A & B   replica to server 6 C & D
> 
> I guess my recommendation is not the best for redundancy and data protection… 
> because I’m concerned with performance, and space, as long as I have 2 copies 
> of the data on different servers then I’m happy.  
> 
> If you care more about performance than space, and want extra data redundancy 
> (more than 2 copies), then use RAID 10 on the nodes, and use gluster replica. 
>  This means you have every byte of data on 4 disks.
> 
> If you care more about space than performance and want extra redundancy use 
> RAID 6, and gluster replica.
> 
> I always recommend gluster replica, because several times I have lost entire 
> servers… and its nice to have the data on more than server.
> 
>> On Jul 4, 2016, at 1:46 PM, Gandalf Corvotempesta 
>>  wrote:
>> 
>> 2016-07-04 19:44 GMT+02:00 Gandalf Corvotempesta
>> :
>>> So, any disk failure would me at least 6TB to be recovered via
>>> network. This mean an high network utilization and as long gluster
>>> doesn't have a dedicated network for replica,
>>> this can slow down client access.
>> 
>> Additionally, using a RAID-0 doesn't give any fault tollerance.
>> My question was for archieving the bast redundancy and data proction
>> available. If I have to use RAID-0 that doesn't protect data, why not
>> removing raid at all ?
> 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[tom]

I would highly stress, regardless of whatever solution you choose - make sure 
you test actual workload performance before going all-in.

In my testing, performance (esp. iops and latency) decreased as I added bricks 
and additional nodes.  Since you have many spindles now, I would encourage you 
to test your workload up to and including the total brick count you ultimately 
expect.  RAID level and whether it’s md, zfs, or hardware isn’t likely to make 
as significant of a performance impact as Gluster and its various clients will. 
 Test failure scenarios and performance characteristics during impairment 
events thoroughly.  Make sure heals happen as you expect, including final 
contents of files modified during an impairment.  If you have many small files 
or directories that will be accessed concurrently, make sure to stress that 
behavior in your testing.

Gluster can be great for targeting availability and distribution at low 
software cost, and I would say as of today at the expense of performance, but 
as with any scale-out NAS there are limitations and some surprises along the 
path.

Good hunting,
-t

> On Jul 4, 2016, at 10:44 AM, Gandalf Corvotempesta 
>  wrote:
> 
> 2016-07-04 19:35 GMT+02:00 Russell Purinton :
>> For 3 servers with 12 disks each, I would do Hardware RAID0 (or madam if you 
>> don’t have a RAID card) of 3 disks.  So four 3-disk RAID0’s per server.
> 
> 3 servers is just to start. We plan to use 5 server in shorter time
> and up to 15 on production.
> 
>> I would set them up as Replica 3 Arbiter 1
>> 
>> server1:/brickA server2:/brickC server3:/brickA
>> server1:/brickB server2:/brickD server3:/brickB
>> server2:/brickA server3:/brickC server1:/brickA
>> server2:/brickB server3:/brickD server1:/brickB
>> server3:/brickA server1:/brickC server2:/brickA
>> server3:/brickB server1:/brickD server2:/brickB
>> 
>> The benefit of this is that you can lose an entire server node (12 disks) 
>> and all of your data is still accessible.   And you get the same space as if 
>> they were all in a RAID10.
>> 
>> If you lose any disk, the entire 3 disk brick will need to be healed from 
>> the replica.   I have 20GbE on each server so it doesn’t take long.   It 
>> copied 20TB in about 18 hours once.
> 
> So, any disk failure would me at least 6TB to be recovered via
> network. This mean an high network utilization and as long gluster
> doesn't have a dedicated network for replica,
> this can slow down client access.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Russell Purinton]

The fault tolerance is provided by Gluster replica translator.

RAID0 to me is preferable to JBOD because you get 3x read performance and 3x 
write performance.   If performance is not a concern, or if you only have 1GbE, 
then it may not matter, and you could just do JBOD with a ton of bricks.

The same method scales to how ever many servers you need… imagine them in a 
ring…

server1 A & B   replica to server 2 C & D
server2 A & B   replica to server 3 C & D
server3 A & B   replica to server 1 C & D

Adding a 4th server?  No problem… you can move the reconfigure the bricks to do
server1 A & B   replica to server 2 C & D
server2 A & B   replica to server 3 C & D
server3 A & B   replica to server 4 C & D
server4 A & B   replica to server 1 C & D

or 5 servers
server1 A & B   replica to server 2 C & D
server2 A & B   replica to server 3 C & D
server3 A & B   replica to server 4 C & D
server4 A & B   replica to server 5 C & D
server5 A & B   replica to server 6 C & D

I guess my recommendation is not the best for redundancy and data protection… 
because I’m concerned with performance, and space, as long as I have 2 copies 
of the data on different servers then I’m happy.  

If you care more about performance than space, and want extra data redundancy 
(more than 2 copies), then use RAID 10 on the nodes, and use gluster replica.  
This means you have every byte of data on 4 disks.

If you care more about space than performance and want extra redundancy use 
RAID 6, and gluster replica.

I always recommend gluster replica, because several times I have lost entire 
servers… and its nice to have the data on more than server.

> On Jul 4, 2016, at 1:46 PM, Gandalf Corvotempesta 
>  wrote:
> 
> 2016-07-04 19:44 GMT+02:00 Gandalf Corvotempesta
> :
>> So, any disk failure would me at least 6TB to be recovered via
>> network. This mean an high network utilization and as long gluster
>> doesn't have a dedicated network for replica,
>> this can slow down client access.
> 
> Additionally, using a RAID-0 doesn't give any fault tollerance.
> My question was for archieving the bast redundancy and data proction
> available. If I have to use RAID-0 that doesn't protect data, why not
> removing raid at all ?

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Gandalf Corvotempesta]

2016-07-04 19:44 GMT+02:00 Gandalf Corvotempesta
:
> So, any disk failure would me at least 6TB to be recovered via
> network. This mean an high network utilization and as long gluster
> doesn't have a dedicated network for replica,
> this can slow down client access.

Additionally, using a RAID-0 doesn't give any fault tollerance.
My question was for archieving the bast redundancy and data proction
available. If I have to use RAID-0 that doesn't protect data, why not
removing raid at all ?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Gandalf Corvotempesta]

2016-07-04 19:35 GMT+02:00 Russell Purinton :
> For 3 servers with 12 disks each, I would do Hardware RAID0 (or madam if you 
> don’t have a RAID card) of 3 disks.  So four 3-disk RAID0’s per server.

3 servers is just to start. We plan to use 5 server in shorter time
and up to 15 on production.

> I would set them up as Replica 3 Arbiter 1
>
> server1:/brickA server2:/brickC server3:/brickA
> server1:/brickB server2:/brickD server3:/brickB
> server2:/brickA server3:/brickC server1:/brickA
> server2:/brickB server3:/brickD server1:/brickB
> server3:/brickA server1:/brickC server2:/brickA
> server3:/brickB server1:/brickD server2:/brickB
>
> The benefit of this is that you can lose an entire server node (12 disks) and 
> all of your data is still accessible.   And you get the same space as if they 
> were all in a RAID10.
>
> If you lose any disk, the entire 3 disk brick will need to be healed from the 
> replica.   I have 20GbE on each server so it doesn’t take long.   It copied 
> 20TB in about 18 hours once.

So, any disk failure would me at least 6TB to be recovered via
network. This mean an high network utilization and as long gluster
doesn't have a dedicated network for replica,
this can slow down client access.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Russell Purinton]

For 3 servers with 12 disks each, I would do Hardware RAID0 (or madam if you 
don’t have a RAID card) of 3 disks.  So four 3-disk RAID0’s per server. 

I would set them up as Replica 3 Arbiter 1

server1:/brickA server2:/brickC server3:/brickA
server1:/brickB server2:/brickD server3:/brickB
server2:/brickA server3:/brickC server1:/brickA
server2:/brickB server3:/brickD server1:/brickB
server3:/brickA server1:/brickC server2:/brickA
server3:/brickB server1:/brickD server2:/brickB

The benefit of this is that you can lose an entire server node (12 disks) and 
all of your data is still accessible.   And you get the same space as if they 
were all in a RAID10.

If you lose any disk, the entire 3 disk brick will need to be healed from the 
replica.   I have 20GbE on each server so it doesn’t take long.   It copied 
20TB in about 18 hours once.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Gandalf Corvotempesta]

2016-07-04 19:25 GMT+02:00 Matt Robinson :
> If you don't trust the hardware raid, then steer clear of raid-6 as mdadm 
> raid 6 is stupidly slow.
> I don't completely trust hardware raid either, but rebuild times should be 
> under a day and in order to lose a raid-6 array you have to lose 3 disks.
> My own systems are hardware raid-6.
> If you're not terribly worried about maximising usable storage, then mdadm 
> raid-10 is your friend.

All of my servers are hardware RAID-6 with 8x300GB SAS 15K (some
servers with 600GB)

A rebuild of a single disk in a 6x600GB SAS RAID-6 takes exactly 22 hours.

This with 15K SAS disks. Now try with 2TB (more than twice the size)
SATA 7200 (less than half speed)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Matt Robinson]

If you don't trust the hardware raid, then steer clear of raid-6 as mdadm raid 
6 is stupidly slow.
I don't completely trust hardware raid either, but rebuild times should be 
under a day and in order to lose a raid-6 array you have to lose 3 disks.
My own systems are hardware raid-6.
If you're not terribly worried about maximising usable storage, then mdadm 
raid-10 is your friend.


> On 4 Jul 2016, at 18:15:26, Gandalf Corvotempesta 
>  wrote:
> 
> 2016-07-04 17:01 GMT+02:00 Matt Robinson :
>> Hi Gandalf,
>> 
>> Are you using hardware raid or mdadm?
>> On high quality hardware raid, a 12 disk raid-6 is pretty solid.  With mdadm 
>> any raid6 (especially with 12 disks) will be rubbish.
> 
> I can use both.
> I don't like very much hardware raid, even high quality. Recently i'm
> having too many issue with hardware raid (like multiple disks kicked
> out with no apparent reasons and virtual-disk failed with data loss)
> 
> A RAID-6 with 12x2TB SATA disks would take days to rebuild, in the
> meanwhile, multiple disks could fail resulting in data loss.
> Yes, gluster is able to recover from this, but I prefere to avoid have
> to resync 24TB of data via networks.
> 
> What about a software RAID-1 ? 6 raid for each gluster nodes and 6
> disks wasted but SATA disks are cheaper.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Joe Julian]

IMHO you use raid for performance reasons and gluster for fault tolerance and 
scale.

On July 4, 2016 7:54:44 AM PDT, Gandalf Corvotempesta 
 wrote:
>No suggestions ?
>Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" <
>gandalf.corvotempe...@gmail.com> ha scritto:
>
>> Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
>> This cluster would be expanded to a maximum of 15 servers in near
>future.
>>
>> What do you suggest, a JBOD or a RAID? Which RAID level?
>>
>> 15 servers with 12 disks/bricks in JBOD are 180 bricks. Is this an
>> acceptable value?
>> Multiple raid6 for each servers? In example, RAID-6 with 6 disks and
>> another RAID-6 with the other 6 disks. I'll loose 4 disks on each
>> servers, performance would be affected and rebuild times would be
>huge
>> (by using 2TB/4TB disks)
>>
>> Any suggestions?
>>
>
>
>
>
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[ML mail]

Hi Gandalf
Not suggesting really here but just mentioning what I am using: I am using an 
HBA adapter with 12 disks so basically JBOD but I am using ZFS and have an 
array of 12 disks in RAIDZ2 (sort of RAID6 but ZFS-style). I am pretty happy 
with that setup so far.
CheersML
 

On Monday, July 4, 2016 4:54 PM, Gandalf Corvotempesta 
 wrote:
 

 No suggestions ?Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" 
 ha scritto:

Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
This cluster would be expanded to a maximum of 15 servers in near future.

What do you suggest, a JBOD or a RAID? Which RAID level?

15 servers with 12 disks/bricks in JBOD are 180 bricks. Is this an
acceptable value?
Multiple raid6 for each servers? In example, RAID-6 with 6 disks and
another RAID-6 with the other 6 disks. I'll loose 4 disks on each
servers, performance would be affected and rebuild times would be huge
(by using 2TB/4TB disks)

Any suggestions?


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

   ___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Matt Robinson]

Hi Gandalf,

Are you using hardware raid or mdadm?
On high quality hardware raid, a 12 disk raid-6 is pretty solid.  With mdadm 
any raid6 (especially with 12 disks) will be rubbish.

Matt.

> On 4 Jul 2016, at 15:54:44, Gandalf Corvotempesta 
>  wrote:
> 
> No suggestions ?
> 
> Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" 
>  ha scritto:
> Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
> This cluster would be expanded to a maximum of 15 servers in near future.
> 
> What do you suggest, a JBOD or a RAID? Which RAID level?
> 
> 15 servers with 12 disks/bricks in JBOD are 180 bricks. Is this an
> acceptable value?
> Multiple raid6 for each servers? In example, RAID-6 with 6 disks and
> another RAID-6 with the other 6 disks. I'll loose 4 disks on each
> servers, performance would be affected and rebuild times would be huge
> (by using 2TB/4TB disks)
> 
> Any suggestions?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] to RAID or not?

[Gandalf Corvotempesta]

No suggestions ?
Il 14 giu 2016 10:01 AM, "Gandalf Corvotempesta" <
gandalf.corvotempe...@gmail.com> ha scritto:

> Let's assume a small cluster made by 3 servers, 12 disks/bricks each.
> This cluster would be expanded to a maximum of 15 servers in near future.
>
> What do you suggest, a JBOD or a RAID? Which RAID level?
>
> 15 servers with 12 disks/bricks in JBOD are 180 bricks. Is this an
> acceptable value?
> Multiple raid6 for each servers? In example, RAID-6 with 6 disks and
> another RAID-6 with the other 6 disks. I'll loose 4 disks on each
> servers, performance would be affected and rebuild times would be huge
> (by using 2TB/4TB disks)
>
> Any suggestions?
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users