Re: [Gluster-users] How to set up a 4 way gluster file system

[Dave Sherohman]

On Fri, Apr 27, 2018 at 07:22:29PM +1200, Thing wrote:
> I have 4 nodes, so a quorum would be 3 of 4.

Nope, gluster doesn't work quite the way you're looking at it.
(Incidentally, I started off with the same expectation that you have.)

When you create a 4-brick replica 2 volume, you don't get a single
cluster with a quorum of 3 out of 4 bricks.  You get two subvolumes,
each of which consists of two mirrored bricks.

Each individual subvolume is susceptible to split-brain if one of the
two bricks in that pair goes down, regardless of how many bricks in
other subvolumes are still up.  Thus, quorum has to be handled on the
subvolume level rather than only being a consideration for the overall
volume as a whole.

One small wrinkle here is that, for calculating quorum, gluster treats
the brick in each pair which was listed first when you created the
volume as "one plus epsilon", so the subvolume will continue to operate
normally if the second brick goes down, but not if the first brick is
missing.

The easy solution to this is to switch from replica 2 to replica 2 +
arbiter.  Arbiter bricks don't need to be nearly as large as data bricks
because they store only file metadata, not file contents, so you can
just scrape up a little spare disk space on two of your boxes, call that
space an arbiter, and run with it.  In my case, I have 10T data bricks
and 100G arbiter bricks; I'm using a total of under 1G across all
arbiter bricks for 3T of data in the volume.

-- 
Dave Sherohman
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to set up a 4 way gluster file system

[Sunil Kumar Heggodu Gopala Acharya]

Hi,

>>gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0

This command will create a distributed-replicate volume(yes you have to opt
'y' at the warning message to get it created). We will have two
distribution legs each containing a replica pair(made of two bricks). When
a file is placed on the volume by a user, it will be placed in one of the
distribution legs which as mentioned earlier will have only two copies.
With replica 2 volumes(volume type: replicate /distributed-replicate) we
might hit split-brain situation. So we recommend replica 3 or arbiter
volume to provide consistency and availability.

Regards,

Sunil kumar Acharya

Senior Software Engineer

Red Hat



T: +91-8067935170 


TRIED. TESTED. TRUSTED. 


On Fri, Apr 27, 2018 at 12:52 PM, Thing  wrote:

> Hi,
>
> I have 4 nodes, so a quorum would be 3 of 4.  The Q is I suppose why does
> the documentation give this command as an example without qualifying it?
>
> SO I am running the wrong command?   I want a "raid10"
>
> On 27 April 2018 at 18:05, Karthik Subrahmanya 
> wrote:
>
>> Hi,
>>
>> With replica 2 volumes one can easily end up in split-brains if there are
>> frequent disconnects and high IOs going on.
>> If you use replica 3 or arbiter volumes, it will guard you by using the
>> quorum mechanism giving you both consistency and availability.
>> But in replica 2 volumes, quorum does not make sense since it needs both
>> the nodes up to guarantee consistency, which costs availability.
>>
>> If you can consider having a replica 3 or arbiter volumes it would be
>> great. Otherwise you can anyway go ahead and continue with the replica 2
>> volume
>> by selecting  *y* for the warning message. It will create the replica 2
>> configuration as you wanted.
>>
>> HTH,
>> Karthik
>>
>> On Fri, Apr 27, 2018 at 10:56 AM, Thing  wrote:
>>
>>> Hi,
>>>
>>> I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
>>> to set these up in a raid 10 which will? give me 2TB useable.  So Mirrored
>>> and concatenated?
>>>
>>> The command I am running is as per documents but I get a warning error,
>>> how do I get this to proceed please as the documents do not say.
>>>
>>> gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
>>> glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
>>> glusterp4:/bricks/brick1/gv0
>>> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
>>> avoid this. See: http://docs.gluster.org/en/lat
>>> est/Administrator%20Guide/Split%20brain%20and%20ways%20to%20
>>> deal%20with%20it/.
>>> Do you still want to continue?
>>>  (y/n) n
>>>
>>> Usage:
>>> volume create  [stripe ] [replica  [arbiter
>>> ]] [disperse []] [disperse-data ] [redundancy ]
>>> [transport ] ?... [force]
>>>
>>> [root@glustep1 ~]#
>>>
>>> thanks
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to set up a 4 way gluster file system

[Thing]

Hi,

I have 4 nodes, so a quorum would be 3 of 4.  The Q is I suppose why does
the documentation give this command as an example without qualifying it?

SO I am running the wrong command?   I want a "raid10"

On 27 April 2018 at 18:05, Karthik Subrahmanya  wrote:

> Hi,
>
> With replica 2 volumes one can easily end up in split-brains if there are
> frequent disconnects and high IOs going on.
> If you use replica 3 or arbiter volumes, it will guard you by using the
> quorum mechanism giving you both consistency and availability.
> But in replica 2 volumes, quorum does not make sense since it needs both
> the nodes up to guarantee consistency, which costs availability.
>
> If you can consider having a replica 3 or arbiter volumes it would be
> great. Otherwise you can anyway go ahead and continue with the replica 2
> volume
> by selecting  *y* for the warning message. It will create the replica 2
> configuration as you wanted.
>
> HTH,
> Karthik
>
> On Fri, Apr 27, 2018 at 10:56 AM, Thing  wrote:
>
>> Hi,
>>
>> I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
>> to set these up in a raid 10 which will? give me 2TB useable.  So Mirrored
>> and concatenated?
>>
>> The command I am running is as per documents but I get a warning error,
>> how do I get this to proceed please as the documents do not say.
>>
>> gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
>> glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
>> glusterp4:/bricks/brick1/gv0
>> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
>> avoid this. See: http://docs.gluster.org/en/lat
>> est/Administrator%20Guide/Split%20brain%20and%20ways%20to%
>> 20deal%20with%20it/.
>> Do you still want to continue?
>>  (y/n) n
>>
>> Usage:
>> volume create  [stripe ] [replica  [arbiter
>> ]] [disperse []] [disperse-data ] [redundancy ]
>> [transport ] ?... [force]
>>
>> [root@glustep1 ~]#
>>
>> thanks
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to set up a 4 way gluster file system

[Karthik Subrahmanya]

Hi,

With replica 2 volumes one can easily end up in split-brains if there are
frequent disconnects and high IOs going on.
If you use replica 3 or arbiter volumes, it will guard you by using the
quorum mechanism giving you both consistency and availability.
But in replica 2 volumes, quorum does not make sense since it needs both
the nodes up to guarantee consistency, which costs availability.

If you can consider having a replica 3 or arbiter volumes it would be
great. Otherwise you can anyway go ahead and continue with the replica 2
volume
by selecting  *y* for the warning message. It will create the replica 2
configuration as you wanted.

HTH,
Karthik

On Fri, Apr 27, 2018 at 10:56 AM, Thing  wrote:

> Hi,
>
> I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
> to set these up in a raid 10 which will? give me 2TB useable.  So Mirrored
> and concatenated?
>
> The command I am running is as per documents but I get a warning error,
> how do I get this to proceed please as the documents do not say.
>
> gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
> glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
> glusterp4:/bricks/brick1/gv0
> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
> avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/
> Split%20brain%20and%20ways%20to%20deal%20with%20it/.
> Do you still want to continue?
>  (y/n) n
>
> Usage:
> volume create  [stripe ] [replica  [arbiter
> ]] [disperse []] [disperse-data ] [redundancy ]
> [transport ] ?... [force]
>
> [root@glustep1 ~]#
>
> thanks
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] How to set up a 4 way gluster file system

[Milind Changire]

On Fri, Apr 27, 2018 at 10:56 AM, Thing  wrote:

> Hi,
>
> I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
> to set these up in a raid 10 which will? give me 2TB useable.  So Mirrored
> and concatenated?
>
> The command I am running is as per documents but I get a warning error,
> how do I get this to proceed please as the documents do not say.
>
> gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
> glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
> glusterp4:/bricks/brick1/gv0
> Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
> avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/
> Split%20brain%20and%20ways%20to%20deal%20with%20it/.
> Do you still want to continue?
>  (y/n) n
>
>
This is not an error. It is a recommendation and an explicit cautionary
step to remind users of the problems using a replica 2.
It is advisable to use a replica 3 instead of a replica 2 to avoid
split-brain conditions.

If you still insist on using a replica 2, you can go ahead and answer a 'y'
to a the "Do you still want to continue ?" question.


> Usage:
> volume create  [stripe ] [replica  [arbiter
> ]] [disperse []] [disperse-data ] [redundancy ]
> [transport ] ?... [force]
>
> [root@glustep1 ~]#
>
> thanks
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Milind
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] How to set up a 4 way gluster file system

[Thing]

Hi,

I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like to
set these up in a raid 10 which will? give me 2TB useable.  So Mirrored and
concatenated?

The command I am running is as per documents but I get a warning error,
how do I get this to proceed please as the documents do not say.

gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See:
http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
.
Do you still want to continue?
 (y/n) n

Usage:
volume create  [stripe ] [replica  [arbiter
]] [disperse []] [disperse-data ] [redundancy ]
[transport ] ?... [force]

[root@glustep1 ~]#

thanks
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users