Re: [Gluster-users] self heal failed, on /

[Mohammed Rafi K C]

On 02/24/2017 11:47 AM, max.degr...@kpn.com wrote:
>
> The version on the server of this specific mount is 3.7.11. The client
> is running version 3.4.2.
>

It is always better to have everything in one version, all clients and
all servers. In this case there is huge gap between the versions, 3.7
and 3.4 .

An additional thing is the code running on 3.4 is replicaV1 code and on
3.7 it v2, meaning there is huge difference to the logic of
replication/healing. So I recommend to keep all the gluster instances to
the same version


~Rafi


>  
>
> There is more to that. This client is actually mounting to volumes
> where the other server is running 3.4.2 as well. What’s your advice,
> update that other server to 3.7.11 (or higher) first? Of start with
> the client update?
>
>  
>
> *Van:*Mohammed Rafi K C [mailto:rkavu...@redhat.com]
> *Verzonden:* vrijdag 24 februari 2017 07:02
> *Aan:* Graaf, Max de; gluster-users@gluster.org
> *Onderwerp:* Re: [Gluster-users] self heal failed, on /
>
>  
>
>  
>
>  
>
> On 02/23/2017 12:18 PM, max.degr...@kpn.com
>  wrote:
>
> Hi,
>
>  
>
> We have a 4 node glusterfs setup that seems to be running without
> any problems. We can’t find any problems with replication or whatever.
>
>  
>
> We also have 4 machines running the glusterfs client. On all 4
> machines we see the following error in the logs at random moments:
>
>  
>
> [2017-02-23 00:04:33.168778] I
> [afr-self-heal-common.c:2869:afr_log_self_heal_completion_status]
> 0-aab-replicate-0:  metadata self heal  is successfully
> completed,   metadata self heal from source aab-client-0 to
> aab-client-1,  aab-client-2,  aab-client-3,  metadata - Pending
> matrix:  [ [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 ] ], on /
>
> [2017-02-23 00:09:34.431089] E
> [afr-self-heal-common.c:2869:afr_log_self_heal_completion_status]
> 0-aab-replicate-0:  metadata self heal  failed,   on /
>
> [2017-02-23 00:14:34.948975] I
> [afr-self-heal-common.c:2869:afr_log_self_heal_completion_status]
> 0-aab-replicate-0:  metadata self heal  is successfully
> completed,   metadata self heal from source aab-client-0 to
> aab-client-1,  aab-client-2,  aab-client-3,  metadata - Pending
> matrix:  [ [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 ] ], on /
>
>  
>
> The content within the glusterfs filesystems is rather static with
> only minor changes on it. This “self heal  failed” is printed
> randomly in the logs on the glusterfs client. It’s printed even at
> moment where nothing has changed within the glusterfs filesystem.
> When it is printed, its never on multiple servers at the same
> time. What we also don’t understand : the error indicates self
> heal failed on root “/”. In the root of this glusterfs mount there
> only 2 folders and no files are ever written at the root level.
>
>  
>
> Any thoughts?
>
>
> From the logs, It looks like an older version of gluster , probably
> 3.5 . Please confirm your glusterfs version. The version is pretty old
> and it may be moved End of Life. And this is AFR v1 , where the latest
> stable version runs with AFRv2.
>
> So I would suggest you to upgrade to a later version may be 3.8 .
>
> If you still want to go with this version, I can give it a try. Let me
> know the version, volume info and volume status. Still I will suggest
> to upgrade ;)
>
>
> Regards
> Rafi KC
>
>
>
>
>  
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>  
>

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Problems restarting gluster

[Mohammed Rafi K C]

On 02/23/2017 03:54 PM, xina towner wrote:
> Hi,
>
> we are using glusterfs with replica 2, we have 16 server nodes and
> around 40 client nodes.
>
> It happens sometimes that they lose connectivity and when we restart
> the node so it can come online again the server kicks us from the
> server and we are unable to login using ssh but the server responds to
> ICMP messages.
>
> I've google a little bit but I'm unable to find any reason why this is
> happening or any fix or workaround. Have you any experienced this
> situation also?
>
> I've found this message:
> http://lists.gluster.org/pipermail/gluster-users.old/2015-February/020635.html
>
> But I can't find any answer.

In straight forward I can't think of any possibilities for dependencies
between glusterfsd process and other userland process. May be somebody
else can help you.

But certainly I would be happy to look into the cause of loosing the
connection between client and bricks. If that helps please get me some
logs and other information like vol info , volume status, and version etc.

Regards
Rafi KC

>
> -- 
> Thanks,
>
> Rubén
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] self heal failed, on /

[max.degraaf]

The version on the server of this specific mount is 3.7.11. The client is 
running version 3.4.2.

There is more to that. This client is actually mounting to volumes where the 
other server is running 3.4.2 as well. What's your advice, update that other 
server to 3.7.11 (or higher) first? Of start with the client update?

Van: Mohammed Rafi K C [mailto:rkavu...@redhat.com]
Verzonden: vrijdag 24 februari 2017 07:02
Aan: Graaf, Max de; gluster-users@gluster.org
Onderwerp: Re: [Gluster-users] self heal failed, on /




On 02/23/2017 12:18 PM, max.degr...@kpn.com wrote:
Hi,

We have a 4 node glusterfs setup that seems to be running without any problems. 
We can't find any problems with replication or whatever.

We also have 4 machines running the glusterfs client. On all 4 machines we see 
the following error in the logs at random moments:

[2017-02-23 00:04:33.168778] I 
[afr-self-heal-common.c:2869:afr_log_self_heal_completion_status] 
0-aab-replicate-0:  metadata self heal  is successfully completed,   metadata 
self heal from source aab-client-0 to aab-client-1,  aab-client-2,  
aab-client-3,  metadata - Pending matrix:  [ [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 
] [ 0 0 0 0 ] ], on /
[2017-02-23 00:09:34.431089] E 
[afr-self-heal-common.c:2869:afr_log_self_heal_completion_status] 
0-aab-replicate-0:  metadata self heal  failed,   on /
[2017-02-23 00:14:34.948975] I 
[afr-self-heal-common.c:2869:afr_log_self_heal_completion_status] 
0-aab-replicate-0:  metadata self heal  is successfully completed,   metadata 
self heal from source aab-client-0 to aab-client-1,  aab-client-2,  
aab-client-3,  metadata - Pending matrix:  [ [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 
] [ 0 0 0 0 ] ], on /

The content within the glusterfs filesystems is rather static with only minor 
changes on it. This "self heal  failed" is printed randomly in the logs on the 
glusterfs client. It's printed even at moment where nothing has changed within 
the glusterfs filesystem. When it is printed, its never on multiple servers at 
the same time. What we also don't understand : the error indicates self heal 
failed on root "/". In the root of this glusterfs mount there only 2 folders 
and no files are ever written at the root level.

Any thoughts?

>From the logs, It looks like an older version of gluster , probably 3.5 . 
>Please confirm your glusterfs version. The version is pretty old and it may be 
>moved End of Life. And this is AFR v1 , where the latest stable version runs 
>with AFRv2.

So I would suggest you to upgrade to a later version may be 3.8 .

If you still want to go with this version, I can give it a try. Let me know the 
version, volume info and volume status. Still I will suggest to upgrade ;)


Regards
Rafi KC









___

Gluster-users mailing list

Gluster-users@gluster.org

http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] How does layout make and store?

[Tahereh Fattahi]

Hi
I want to know how the layout of directories does store? Is everything just
in trusted.glusterfs.dht? or something else has role in this subject?
another question is about making the layout, does layout make in client
side or server side? for example the trusted.glusterfs.dht get value in
client or server? I had thought that client should do that but the field
list[].start and list[].stop in layout structure does not get value in
mkdir but after mkdir this attribute has value in server side
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] self heal failed, on /

[Mohammed Rafi K C]

On 02/23/2017 12:18 PM, max.degr...@kpn.com wrote:
>
> Hi,
>
>  
>
> We have a 4 node glusterfs setup that seems to be running without any
> problems. We can’t find any problems with replication or whatever.
>
>  
>
> We also have 4 machines running the glusterfs client. On all 4
> machines we see the following error in the logs at random moments:
>
>  
>
> [2017-02-23 00:04:33.168778] I
> [afr-self-heal-common.c:2869:afr_log_self_heal_completion_status]
> 0-aab-replicate-0:  metadata self heal  is successfully completed,  
> metadata self heal from source aab-client-0 to aab-client-1, 
> aab-client-2,  aab-client-3,  metadata - Pending matrix:  [ [ 0 0 0 0
> ] [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 ] ], on /
>
> [2017-02-23 00:09:34.431089] E
> [afr-self-heal-common.c:2869:afr_log_self_heal_completion_status]
> 0-aab-replicate-0:  metadata self heal  failed,   on /
>
> [2017-02-23 00:14:34.948975] I
> [afr-self-heal-common.c:2869:afr_log_self_heal_completion_status]
> 0-aab-replicate-0:  metadata self heal  is successfully completed,  
> metadata self heal from source aab-client-0 to aab-client-1, 
> aab-client-2,  aab-client-3,  metadata - Pending matrix:  [ [ 0 0 0 0
> ] [ 0 0 0 0 ] [ 0 0 0 0 ] [ 0 0 0 0 ] ], on /
>
>  
>
> The content within the glusterfs filesystems is rather static with
> only minor changes on it. This “self heal  failed” is printed randomly
> in the logs on the glusterfs client. It’s printed even at moment where
> nothing has changed within the glusterfs filesystem. When it is
> printed, its never on multiple servers at the same time. What we also
> don’t understand : the error indicates self heal failed on root “/”.
> In the root of this glusterfs mount there only 2 folders and no files
> are ever written at the root level.
>
>  
>
> Any thoughts?
>

>From the logs, It looks like an older version of gluster , probably 3.5
. Please confirm your glusterfs version. The version is pretty old and
it may be moved End of Life. And this is AFR v1 , where the latest
stable version runs with AFRv2.

So I would suggest you to upgrade to a later version may be 3.8 .

If you still want to go with this version, I can give it a try. Let me
know the version, volume info and volume status. Still I will suggest to
upgrade ;)


Regards
Rafi KC



>  
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Charm now supports ZFS bricks

[Mohammed Rafi K C]

Great effort. Kudos to the team.


Regards

Rafi KC


On 02/23/2017 07:12 PM, chris holcombe wrote:
> Hey Gluster Community!
>
> I wanted to announce that I have built support for ZFS bricks into the
> Gluster charm: https://github.com/cholcombe973/gluster-charm.  If anyone
> wants to give it a spin and provide feedback I would be overjoyed :).
>
> Thanks,
> Chris
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] detecting replication issues

[Mohammed Rafi K C]

On 02/23/2017 11:12 PM, Joseph Lorenzini wrote:
> Hi all,
>
> I have a simple replicated volume with a replica count of 3. To ensure
> any file changes (create/delete/modify) are replicated to all bricks,
> I have this setting in my client configuration.
>
>  volume gv0-replicate-0
> type cluster/replicate
> subvolumes gv0-client-0 gv0-client-1 gv0-client-2
> end-volume
>
> And that works as expected. My question is how one could detect if
> this was not happening which could poise a severe problem with data
> consistency and replication. For example, those settings could be
> omitted from the client config and then the client will only write
> data to one brick and all kinds of terrible things will start
> happening. I have not found a way the gluster volume cli to detect
> when that kind of problem is occurring. For example gluster volume
> heal  info does not detect this problem. 
>
> Is there any programmatic way to detect when this problem is occurring?
>

I couldn't understand how you will end up in this situation. There is
only one possibility (assuming there is no bug :) ), ie you changed the
client graph in a way that there is only one subvolume to replica server.

To check that the simply way is, there is a xlator called meta, which
provides meta data information through mount point, similiar to linux
proc file system. So you can check the active graph through meta and see
the number of subvolumes for replica xlator

for example : the directory   /.meta/graphs/active/-replicate-0/subvolumes will have
entries for each replica clients , so in your case you should see 3
directories.


Let me know if this helps.

Regards
Rafi KC


> Thanks,
> Joe
>
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and ZFS: how much RAM?

[Darrell Budic]

What are you looking at to determine the hit rate? l2_hits & l2_misses, or hits 
& misses?

I’ve got a small cluster with 16G boxes and I’m about on those numbers for the 
l2arc, my bigger group with dedicated storage servers uses more, but I’d like 
to do more analysis on it and see how effective it is.


> On Feb 23, 2017, at 4:53 AM, David Gossage  
> wrote:
> 
> On Thu, Feb 23, 2017 at 3:57 AM, Gandalf Corvotempesta 
> mailto:gandalf.corvotempe...@gmail.com>> 
> wrote:
> How much RAM is suggested for gluster with ZFS (no dedup) ?
> 
> Are 16GB enough with an SSD L2ARC?
> 
> 8gb for the arc, 8gb for gluster and OS.
> 
> That's what my systems run though each of my nodes only has @3TB of storage.  
> Current tests among zfs used as VM storage seems to be that l2arc sits almost 
> wholly unused though.  I removed mine since it had such a very low hit rate. 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://lists.gluster.org/mailman/listinfo/gluster-users 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] detecting replication issues

[Joseph Lorenzini]

Hi all,

I have a simple replicated volume with a replica count of 3. To ensure any
file changes (create/delete/modify) are replicated to all bricks, I have
this setting in my client configuration.

 volume gv0-replicate-0
type cluster/replicate
subvolumes gv0-client-0 gv0-client-1 gv0-client-2
end-volume

And that works as expected. My question is how one could detect if this was
not happening which could poise a severe problem with data consistency and
replication. For example, those settings could be omitted from the client
config and then the client will only write data to one brick and all kinds
of terrible things will start happening. I have not found a way the gluster
volume cli to detect when that kind of problem is occurring. For example
gluster volume heal  info does not detect this problem.

Is there any programmatic way to detect when this problem is occurring?

Thanks,
Joe
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster Charm now supports ZFS bricks

[chris holcombe]

Hey Gluster Community!

I wanted to announce that I have built support for ZFS bricks into the
Gluster charm: https://github.com/cholcombe973/gluster-charm.  If anyone
wants to give it a spin and provide feedback I would be overjoyed :).

Thanks,
Chris
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and ZFS: how much RAM?

[Lindsay Mathieson]

On 23/02/2017 9:49 PM, Gandalf Corvotempesta wrote:

Anyway, is possible to use the same ZIL partition for multiple
bricks/zfs vdevs ?


I presume you mean slog rather than zil :)


The slog is per pool and applies to all vdevs in the pool.

--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and ZFS: how much RAM?

[Gandalf Corvotempesta]

2017-02-23 11:53 GMT+01:00 David Gossage :
> That's what my systems run though each of my nodes only has @3TB of storage.
> Current tests among zfs used as VM storage seems to be that l2arc sits
> almost wholly unused though.  I removed mine since it had such a very low
> hit rate.

I have to use an SSD for ZIL, the same SSD could also be used for
L2ARC. I don't think would give me any advantage to remove that at
all.

Anyway, is possible to use the same ZIL partition for multiple
bricks/zfs vdevs ? (i'll use one vdev per brick, but I would like to
share the same SSD with all vdevs without creating multiple partitions
in the same SSD)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and ZFS: how much RAM?

[David Gossage]

On Thu, Feb 23, 2017 at 3:57 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> How much RAM is suggested for gluster with ZFS (no dedup) ?
>
> Are 16GB enough with an SSD L2ARC?
>
> 8gb for the arc, 8gb for gluster and OS.
>

That's what my systems run though each of my nodes only has @3TB of
storage.  Current tests among zfs used as VM storage seems to be that l2arc
sits almost wholly unused though.  I removed mine since it had such a very
low hit rate.


> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster and ZFS: how much RAM?

[Lindsay Mathieson]

On 23/02/2017 7:57 PM, Gandalf Corvotempesta wrote:

How much RAM is suggested for gluster with ZFS (no dedup) ?

Are 16GB enough with an SSD L2ARC?

8gb for the arc, 8gb for gluster and OS.


I have a 8GB limitation for my ZFS arc. Gluster (server+client) only 
seems to use about 3-4GB, so you should be fine with 16GB



For most people, running KVM VM's over zfs direct or via gluster over 
zfs, the l2arc doesn't seem to be much use, very poor hit ratio, less 
than 6%


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Problems restarting gluster

[xina towner]

Hi,

we are using glusterfs with replica 2, we have 16 server nodes and around
40 client nodes.

It happens sometimes that they lose connectivity and when we restart the
node so it can come online again the server kicks us from the server and we
are unable to login using ssh but the server responds to ICMP messages.

I've google a little bit but I'm unable to find any reason why this is
happening or any fix or workaround. Have you any experienced this situation
also?

I've found this message:
http://lists.gluster.org/pipermail/gluster-users.old/2015-February/020635.html

But I can't find any answer.

-- 
Thanks,

Rubén
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster and ZFS: how much RAM?

[Gandalf Corvotempesta]

How much RAM is suggested for gluster with ZFS (no dedup) ?

Are 16GB enough with an SSD L2ARC?

8gb for the arc, 8gb for gluster and OS.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users