Re: [Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Vikas Gorur]

Alex Attarian wrote:
Thanks Vikas, but how does it know that it's the first access for the 
file? Also what happens when somehow a file got deleted or corrupted 
on one of the servers? At what point will gluster replicate that file 
again?


Before any operation can be performed on a file by an application, the 
kernel sends a 'lookup' call
to the filesystem. The 'lookup' is what I meant by first access. 
Self-heal of a file also happens on a
lookup if replicate detects that there's an inconsistency among the 
subvolumes.


Vikas


___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Alex Attarian]

Thanks Vikas, but how does it know that it's the first access for the file?
Also what happens when somehow a file got deleted or corrupted on one of the
servers? At what point will gluster replicate that file again?

Alex

how does it know

On Tue, Feb 2, 2010 at 12:56 AM, Vikas Gorur  wrote:

> Alex Attarian wrote:
>
>> I'll give that a try Gordan! Thanks for the info!
>> My other question is does gluster perform a read for the extended
>> attributes on all servers regardless read-subvolume?
>> This is what my tcpdump shows:
>>
> [snip]
>
>
>  Does it have to read the trusted.afr.* attributes from every server every
>> time it reads a file?
>>
> Yes, the extended attributes are read from all servers on the first access
> to the file (i.e., lookup).
> It is not done on every read ().
>
> As Gordon said, this is done to gather the metadata needed to decide
> whether a self-heal
> needs to be done.
>
> Vikas
> --
> Engineer - Gluster, Inc.
>
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Vikas Gorur]

Alex Attarian wrote:
I'll give that a try Gordan! Thanks for the info! 

My other question is does gluster perform a read for the extended 
attributes on all servers regardless read-subvolume?

This is what my tcpdump shows:

[snip]

Does it have to read the trusted.afr.* attributes from every server 
every time it reads a file?
Yes, the extended attributes are read from all servers on the first 
access to the file (i.e., lookup).

It is not done on every read ().

As Gordon said, this is done to gather the metadata needed to decide 
whether a self-heal

needs to be done.

Vikas
--
Engineer - Gluster, Inc.


___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Gordan Bobic]

Alex Attarian wrote:
I'll give that a try Gordan! Thanks for the info! 

My other question is does gluster perform a read for the extended 
attributes on all servers regardless read-subvolume?

This is what my tcpdump shows:
,3..h...:O5*.L. 
..+c...c..Dz/production/public_images/15932/images/JetLease_Aircraft_Wanted_For_Lease_Oct_09.gif.JetLease_Aircraft_Wanted_For_Lease_Oct_09.gif.glusterfs.open-fd-count.0.trusted.afr.gfs3.12.trusted.afr.gfs2.12.trusted.afr.gfs1.12.
09:17:23.498739 IP 192.168.27.26.6996 > 192.168.13.14.1022: P 
10821:11057(236) ack 13336 win 1525 

..]...
h...,3..:O.*.L. 
..@c...ckfgxj...j..pglusterfs.open-fd-count.0.trusted.afr.gfs2.trusted.afr.gfs1.
09:17:23.511931 IP 192.168.13.14.1022 > 192.168.27.26.6996: P 
13336:13625(289) ack 11057 win 1525 

..J..
,3..h...:O.*.L. 
..+c...c..Dz/production/public_images/15932/images/Jet_1Hawker_Multi_July_09.gif.Jet_1Hawker_Multi_July_09.gif.glusterfs.open-fd-count.0.trusted.afr.gfs3.12.trusted.afr.gfs2.12.trusted.afr.gfs1.12.
09:17:23.512218 IP 192.168.27.26.6996 > 192.168.13.14.1022: P 
11057:11293(236) ack 13625 win 1525 

..
h...,3..:O.*.L. 
..A[...g...c...cKBp.Jw..Jw.pglusterfs.open-fd-count.0.trusted.afr.gfs2.trusted.afr.gfs1.
09:17:23.525473 IP 192.168.13.14.1022 > 192.168.27.26.6996: P 
13625:13944(319) ack 11293 win 1525 

..J-..


Does it have to read the trusted.afr.* attributes from every server 
every time it reads a file?


In short - yes, AFAIK. Healing is done when files are accessed, thus the 
metadata checks have to be done across all the servers when a file is 
accessed to verify that they are all in sync.


The end result is that this ends up being the dominant performance 
factor for small I/Os and negligible for large I/O. So if your files are 
small and many (e.g. maildir), performance will be poor. But then again 
all cluster file systems suffer from that.


Gordan


___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Alex Attarian]

I'll give that a try Gordan! Thanks for the info!

My other question is does gluster perform a read for the extended attributes
on all servers regardless read-subvolume?
This is what my tcpdump shows:
,3..h...:O5*.L.
..+c...c..Dz/production/public_images/15932/images/JetLease_Aircraft_Wanted_For_Lease_Oct_09.gif.JetLease_Aircraft_Wanted_For_Lease_Oct_09.gif.glusterfs.open-fd-count.0.trusted.afr.gfs3.12.trusted.afr.gfs2.12.trusted.afr.gfs1.12.
09:17:23.498739 IP 192.168.27.26.6996 > 192.168.13.14.1022: P
10821:11057(236) ack 13336 win 1525 
..]...
h...,3..:O.*.L.
..@c...ckfgxj...j..pglusterfs.open-fd-count.0.trusted.afr.gfs2.trusted.afr.gfs1.
09:17:23.511931 IP 192.168.13.14.1022 > 192.168.27.26.6996: P
13336:13625(289) ack 11057 win 1525 
..J..
,3..h...:O.*.L.
..+c...c..Dz/production/public_images/15932/images/Jet_1Hawker_Multi_July_09.gif.Jet_1Hawker_Multi_July_09.gif.glusterfs.open-fd-count.0.trusted.afr.gfs3.12.trusted.afr.gfs2.12.trusted.afr.gfs1.12.
09:17:23.512218 IP 192.168.27.26.6996 > 192.168.13.14.1022: P
11057:11293(236) ack 13625 win 1525 
..
h...,3..:O.*.L.
..A[...g...c...cKBp.Jw..Jw.pglusterfs.open-fd-count.0.trusted.afr.gfs2.trusted.afr.gfs1.
09:17:23.525473 IP 192.168.13.14.1022 > 192.168.27.26.6996: P
13625:13944(319) ack 11293 win 1525 
..J-..


Does it have to read the trusted.afr.* attributes from every server every
time it reads a file?

Thanks,
Alex


On Sun, Jan 31, 2010 at 10:42 AM, Gordan Bobic  wrote:

> I seem to remember something like this being broken was mentioned on the
> list a while back. I wouldn't even bother asking about issues like this
> before trying the latest stable version (2.0.9).
>
> Gordan
>
> Alex Attarian wrote:
>
>> Is anyone willing to look at this and maybe provide some answers? Is there
>> a magic word I need to use to get someone to answer? It always looks like
>> certain people get attention from the devs right away, but some of us just
>> get left off hanging with no answers.
>>
>> Alex
>>
>> On Fri, Jan 29, 2010 at 2:55 PM, Alex Attarian > u2sas...@gmail.com>> wrote:
>>
>>Hi guys,
>>
>>I have 3 server AFR setup, two servers are in a local net and one
>>server is remote in another location using Gluster 2.0.6. I have
>>also 5 local clients that are connected to all 3 servers. Reads are
>>mostly slow because of the connection to the 3rd server, however I
>>thought enabling read-subvolume and saying to read from one of the
>>local nodes would solve that. However, I still see reads going to
>>the 3rd node, although I specified read-subvolume local-node2.
>>When I disable the 3rd server on all clients inside AFR, reads
>>become very fast, which makes me assume that it's the reads to the
>>remote server that is making everything slow.
>>
>>Is it not working or broken in 2.0.6?
>>
>>Thanks,
>>Alex
>>
>>
>>
>> 
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>
> ___
> Gluster-devel mailing list
> Gluster-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Gordan Bobic]

I seem to remember something like this being broken was mentioned on the 
list a while back. I wouldn't even bother asking about issues like this 
before trying the latest stable version (2.0.9).


Gordan

Alex Attarian wrote:
Is anyone willing to look at this and maybe provide some answers? Is 
there a magic word I need to use to get someone to answer? It always 
looks like certain people get attention from the devs right away, but 
some of us just get left off hanging with no answers.


Alex

On Fri, Jan 29, 2010 at 2:55 PM, Alex Attarian > wrote:


Hi guys,

I have 3 server AFR setup, two servers are in a local net and one
server is remote in another location using Gluster 2.0.6. I have
also 5 local clients that are connected to all 3 servers. Reads are
mostly slow because of the connection to the 3rd server, however I
thought enabling read-subvolume and saying to read from one of the
local nodes would solve that. However, I still see reads going to
the 3rd node, although I specified read-subvolume local-node2.
When I disable the 3rd server on all clients inside AFR, reads
become very fast, which makes me assume that it's the reads to the
remote server that is making everything slow.

Is it not working or broken in 2.0.6?

Thanks,
Alex





___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel




___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel

[Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Alex Attarian]

Is anyone willing to look at this and maybe provide some answers? Is there a
magic word I need to use to get someone to answer? It always looks like
certain people get attention from the devs right away, but some of us just
get left off hanging with no answers.

Alex

On Fri, Jan 29, 2010 at 2:55 PM, Alex Attarian  wrote:

> Hi guys,
>
> I have 3 server AFR setup, two servers are in a local net and one server is
> remote in another location using Gluster 2.0.6. I have also 5 local clients
> that are connected to all 3 servers. Reads are mostly slow because of the
> connection to the 3rd server, however I thought enabling read-subvolume and
> saying to read from one of the local nodes would solve that. However, I
> still see reads going to the 3rd node, although I specified read-subvolume
> local-node2.
> When I disable the 3rd server on all clients inside AFR, reads become very
> fast, which makes me assume that it's the reads to the remote server that is
> making everything slow.
>
> Is it not working or broken in 2.0.6?
>
> Thanks,
> Alex
>
___
Gluster-devel mailing list
Gluster-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel