Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Dev Sidious]

Unfortunately, gandalf is precisely right with the point he made on data
consistency in GlusterFS.

> If gluster isn't able to ensure data consistency when doing it's
> primary role, scaling up a storage, i'm sorry but it can't be
> considered "enterprise" ready or production ready.

In my short experience with GlusterFS I have known it to fail PRECISELY
on data consistency (data representation consistency to be more
precise). Namely:

a) files partially or not at all replicated due to
b) errors such as: "Transport endpoint not connected"

with more or less random frequency.

I solved all these by disabling SSL. Since I disabled SSL, the system
APPEARS to be reliable.

To me, a system exhibiting such a behavior is not a solid system.

If it's "production ready" or not, now that's a more subjective topic
and I will leave it to the arm chair computer scientists and the
philosophers.




On 3/19/2017 12:53 AM, Krutika Dhananjay wrote:
> 
> 
> On Sat, Mar 18, 2017 at 11:15 PM, Gandalf Corvotempesta
>  > wrote:
> 
> Krutika, it wasn't an attack directly to you.
> It wasn't an attack at all.
> 
> 
> Gluster is a "SCALE-OUT" software defined storage, the folllowing is
> wrote in the middle of the homepage:
> "GlusterFS is a scalable network filesystem"
> 
> So, scaling a cluster is one of the primary goal of gluster.
> 
> A critical bug that prevent gluster from being scaled without loosing
> data was discovered 1 year ago, and took 1 year to be fixed. 
> 
> 
> If gluster isn't able to ensure data consistency when doing it's
> primary role, scaling up a storage, i'm sorry but it can't be
> considered "enterprise" ready or production ready.
> 
> 
> That's not entirely true. VM use-case is just one of the many workloads
> users
> use Gluster for. I think I've clarified this before. The bug was in
> dht-shard interaction.
> And shard is *only* supported in VM use-case as of today. This means that
> scaling out has been working fine on all but the VM use-case.
> That doesn't mean that Gluster is not production-ready. At least users
> who've deployed Gluster
> in non-VM use-cases haven't complained of add-brick not working in the
> recent past.
> 
> 
> -Krutika
>  
> 
> Maybe SOHO for small offices or home users, but in enterprises, data
> consistency and reliability is the most important thing and gluster
> isn't able to guarantee this even
> doing a very basic routine procedure that should be considered as the
> basis of the whole gluster project (as wrote on gluster's homepage)
> 
> 
> 2017-03-18 14:21 GMT+01:00 Krutika Dhananjay  >:
> >
> >
> > On Sat, Mar 18, 2017 at 3:18 PM, Gandalf Corvotempesta
> >  > wrote:
> >>
> >> 2017-03-18 2:09 GMT+01:00 Lindsay Mathieson
> >:
> >> > Concerning, this was supposed to be fixed in 3.8.10
> >>
> >> Exactly. https://bugzilla.redhat.com/show_bug.cgi?id=1387878
> 
> >> Now let's see how much time they require to fix another CRITICAL bug.
> >>
> >> I'm really curious.
> >
> >
> > Hey Gandalf!
> >
> > Let's see. There have been plenty of occasions where I've sat and
> worked on
> > users' issues on weekends.
> > And then again, I've got a life too outside of work (or at least I'm
> > supposed to), you know.
> > (And hey you know what! Today is Saturday and I'm sitting here and
> > responding to your mail and collecting information
> > on Mahdi's issue. Nobody asked me to look into it. I checked the
> mail and I
> > had a choice to ignore it and not look into it until Monday.)
> >
> > Is there a genuine problem Mahdi is facing? Without a doubt!
> >
> > Got a constructive feedback to give? Please do.
> > Do you want to give back to the community and help improve
> GlusterFS? There
> > are plenty of ways to do that.
> > One of them is testing out the releases and providing feedback.
> Sharding
> > wouldn't have worked today, if not for Lindsay's timely
> > and regular feedback in several 3.7.x releases.
> >
> > But this kind of criticism doesn't help.
> >
> > Also, spending time on users' issues is only one of the many
> > responsibilities we have as developers.
> > So what you see on mailing lists is just the tip of the iceberg.
> >
> > I have personally tried several times to recreate the add-brick
> bug on 3
> > machines I borrowed from Kaleb. I haven't had success in
> recreating it.
> > Reproducing VM-related bugs, in my experience, wasn't easy. I
> don't use
> > Proxmox. 

Re: [Gluster-users] File operation failure on simple distributed volume

[Dev Sidious]

Hi,

I initially wanted to post this separately, but this thread seems on
subject, so I'm chiming in here.

I have a distributed-replicated volume with 4 bricks and I too have
observed file operation "weirdness" with GlusterFS. Examples:

a) I would create a folder or copy a file on the volume locally from
server A, then login to server B and try to delete it. Error =>
"Transport endpoint is not connected".

What fixes it: server B reboot. Sometimes multiple reboots of the server
in cause are needed.

b) I would copy multiple config files to the distributed volume from
server A, then login on server B and C and have everything available
there, but on server D, some (not all) of the files are missing.

What fixes it: server D reboot and recopying of the files on the volume.

Sometimes this behavior becomes apparent after I sequentially reboot all
my servers. At other times I work on the servers, then take a break
leaving everything in order and come back a few days later just to bump
into this behavior.

Sometimes only one server is affected. At other times multiple servers
are affected.

The output of "gluster volume info", "gluster pool list", "gluster
volume status" shows everything is in order all the time. I have never
observed anything out of ordinary there.

I need to add that none of my servers use separate partitions for
Gluster, everything resides on the main system partition.

Besides this current thread, I found another one from June 2016 on the
same subject:

http://lists.gluster.org/pipermail/gluster-devel/2016-June/049933.html

Any ideas ?


Some information retrieved while experiencing the problems mentioned:
-

[root@wasa volume-r2-d4]# rm -rf test/
rm: cannot remove ‘test/’: Transport endpoint is not connected

[root@wasa ~]# glusterd -V
glusterfs 3.8.9 built on Feb 13 2017 10:03:47

[root@wasa volume-r2-d4]# gluster volume info

Volume Name: volume-r2-d4
Type: Distributed-Replicate
Volume ID: e6a6e0f0-c102-4f32-abe9-25e9c21ff9c5
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: wasa:/glusterfs/volume-r2-d4
Brick2: wasb:/glusterfs/volume-r2-d4
Brick3: wasc:/glusterfs/volume-r2-d4
Brick4: wasd:/glusterfs/volume-r2-d4
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
client.ssl: on
server.ssl: on
auth.ssl-allow: wasa,wasb,wasc,wasd

[root@wasa volume-r2-d4]# gluster pool list
UUIDHostnameState
7316377c-8535-4c5c-a4aa-d355eb85c1e3wasbConnected
6d095b48-5ea5-41f7-a8fc-a8ece87f969ewascConnected
0ebb88c0-c5f7-4683-8404-cb152040d59awasdConnected
6a0594da-11bd-4f84-b8cb-94da9597858alocalhost   Connected

[root@wasa volume-r2-d4]# gluster volume status volume-r2-d4
Status of volume: volume-r2-d4
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick wasa:/glusterfs/volume-r2-d4  49155 0  Y
4541
Brick wasb:/glusterfs/volume-r2-d4  49154 0  Y
4911
Brick wasc:/glusterfs/volume-r2-d4  49154 0  Y
3907
Brick wasd:/glusterfs/volume-r2-d4  49154 0  Y
3724
Self-heal Daemon on localhost   N/A   N/AY
5251
Self-heal Daemon on wasdN/A   N/AY
4858
Self-heal Daemon on wascN/A   N/AY
5057
Self-heal Daemon on wasbN/A   N/AY
5096

Task Status of Volume volume-r2-d4
--
There are no active volume tasks

[root@wasa volume-r2-d4]# gluster volume heal volume-r2-d4 info
Brick wasa:/glusterfs/volume-r2-d4
Status: Connected
Number of entries: 0

Brick wasb:/glusterfs/volume-r2-d4
Status: Connected
Number of entries: 0

Brick wasc:/glusterfs/volume-r2-d4
Status: Connected
Number of entries: 0

Brick wasd:/glusterfs/volume-r2-d4
Status: Connected
Number of entries: 0


On 3/7/2017 4:23 AM, yonex wrote:
> Hi Rafi
> 
> Sorry for the late. Though I eventually could not have reproduced the
> problem out of the production environment, I will be able to run the
> debug build as part of the production if it does not occur a
> performance issue. I would like you to give me a guide about the debug
> build. By the way, before that, as it would be helpful to update
> glusterfs from 3.8.5 to 3.8.9, I am going to do this.
> 
> Regards
> 
> Yonex
> 
> 2017-02-17 15:03 GMT+09:00 Mohammed Rafi K C :
>> Hi Yonex
>>
>> Recently Poornima has fixed one corruption issue with upcall, which
>> seems unlikely the cause of the issue, given that you are running fuse
>> clients. Even then I would like to give you a debug build including the
>> fix [1] and adding additional logs.
>>
>> Will you 

Re: [Gluster-users] Swap space requirements

[Dev Sidious]

ERATA:

less than 4 GB RAM.

RAM < 4 GB , SWAP = RAM * 2
RAM > 4 GB, SWAP = RAM + 2 GB

On 2/7/2017 5:37 PM, Cedric Lemarchand wrote:
> Hum ... I would say that ram is really expensive right now ! 
> In response to the OP, an old rule was to provision double ram as swap,
> actually you could put the half, or nothing.  
> From my point of view, swap only helps in the way it retards OOM killer
> process and as a sort of warning role because it slows a lot the whole OS. 
> 
> Cheers
> 
> --
> Cédric Lemarchand
> 
> Le 7 févr. 2017 à 17:11, Gambit15  > a écrit :
> 
>> Gluster doesn't "require" swap any more than any other service, and
>> with the price of RAM today, most admins should even consider removing
>> swap altogether.
>>
>> D
>>
>> On 7 February 2017 at 10:56, Mark Connor > > wrote:
>>
>> I am planning in deploying about 18 bricks of about 50 TB bricks
>> each spanning 8-10 servers. My servers are high end servers with
>> 128gb each. I have searched and cannot find any detail on swap
>> partition requirements for the latest gluster server.  Can anyone
>> offer me some advice?
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>> 
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-users
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
> 



signature.asc
Description: OpenPGP digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Distributed volumes

[Dev Sidious]

Hi,

New to Gluster too. I welcome the more experienced users to correct me
if I'm wrong.

Based on some quick tests in my environment, it works like this:

a) The following creates a replicated (replica count = 2),
non-distributed volume

gluster volume create replicated_but_not_distributed replica 2
host1:/GlusterFS/replicated_but_not_distributed
host2:/GlusterFS/replicated_but_not_distributed

[user@whatever]# gluster volume info replicated_but_not_distributed

Volume Name: replicated_but_not_distributed
Type: Replicate
Volume ID: blah-bal-blah-blah-blah
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: host1:/GlusterFS/replicated_but_not_distributed
Brick2: host2:/GlusterFS/replicated_but_not_distributed
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet


b) The following creates a distributed, non-replicated volume. Note that
I omitted the "replica" directive:

gluster volume create distributed_but_not_replicated
host1:/GlusterFS/distributed_but_not_replicated
host2:/GlusterFS/distributed_but_not_replicated

[user@whatever]# gluster volume info distributed_but_not_replicated

Volume Name: distributed_but_not_replicated
Type: Distribute
Volume ID: blah-bal-blah-blah-blah
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: host1:/GlusterFS/distributed_but_not_replicated
Brick2: host2:/GlusterFS/distributed_but_not_replicated
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

c) The following creates a replicated AND distributed volume as shown
here
(https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/sect-User_Guide-Setting_Volumes-Distributed_Replicated.html):

gluster volume create replicated_and_distributed replica 2
host1:/GlusterFS/replicated_and_distributed
host2:/GlusterFS/replicated_and_distributed
host3:/GlusterFS/replicated_and_distributed
host4:/GlusterFS/replicated_and_distributed

I don't have 2 other nodes online at this moment to paste the output of
"gluster volume info" for this one.

I hope this helps.

On 2/8/2017 4:12 AM, Dave Fan wrote:
> Hi,
> 
> I'm new to Gluster so a very basic question. Are all volumes distributed
> by default? Is there a switch to turn this feature on/off?
> 
> I ask this because in an intro to Gluster I saw "Replicated Volume" and
> "Distributed Replicated Volume". Is the first type, "Replicated Volume",
> not distributed?
> 
> Many thanks,
> Dave
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
> 



signature.asc
Description: OpenPGP digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] GlusterFS & Disk configuration

[Dev Sidious]

Hello,

The Quick Start Guide suggests having "At least two virtual disks, one
for the OS installation, and one to be used to serve GlusterFS storage
(sdb)".
(https://gluster.readthedocs.io/en/latest/Quick-Start-Guide/Quickstart/)

For my particular needs this suggestion is less than optimal, therefore
I created the storage directory on the same disk. Gluster whined about
it and I had to use "force".

I would like  to understand what the implications are in this case, but
I haven't been able to find much material on the subject.

Any input would be most welcome.

Thank you.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users