Re: [Gluster-users] Replicate over WAN?
Am 07.05.2010 09:43, schrieb Diego Zuccato: On 05/05/2010 20:18, Vikas Gorur wrote: 2) "readdir" (ls) is always sent to the first subvolume. This is necessary to ensure consistent inode numbers. Uhm... Couldn't the same result be achieved storing a "virtual inode number" in an attribute? So that it gets replicated with the rest of the data and it makes possible to have the first subvolume always local... Maybe I don't understand your proposal but I don't see how this helps. Remember: Servers don't know about clients and as long as there is no way for SERVERS to notify the clients about writes from others (i.e. server side cache invalidation), the client has to check before each and all operations. cheers Paul ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replicate over WAN?
On 05/05/2010 20:18, Vikas Gorur wrote: 2) "readdir" (ls) is always sent to the first subvolume. This is necessary to ensure consistent inode numbers. Uhm... Couldn't the same result be achieved storing a "virtual inode number" in an attribute? So that it gets replicated with the rest of the data and it makes possible to have the first subvolume always local... I understand tht it could lead to possible problems (like "how do I generate an inode number if the master node is missing"), but it could open to the "replicate writes, local reads" that many people are requesting... The scenario to think about is a firm w/ a remote office connected via a VPN -- if you can cut nearly all the read traffic from the VPN, then you see a great boost in performance. Or maybe I missed something... -- Diego Zuccato Servizi Informatici Dip. di Astronomia - Università di Bologna Via Ranzani, 1 - 40126 Bologna - Italy tel.: +39 051 20 95786 mail: diego.zucc...@unibo.it -- LA RICERCA C’È E SI VEDE: 5 per mille all'Università di Bologna - C.F.: 80007010376 http://www.unibo.it/Vademecum5permille.htm Questa informativa è inserita in automatico dal sistema al fine esclusivo della realizzazione dei fini istituzionali dell’ente. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replicate over WAN?
Count Zero, Just to clarify, the three patches were not "released", they will make into a future release after proper testing. They are not developed to fix WAN issues per se, just to help with small files and slow "ls" kind of problems which are seen with huge numbers of files or huge number of small files or else deep directories. As a side effect, they *may* fix some of your WAN issues :-). Let me know how things go. Please don't use in production. Regards, Tejas. - Original Message - From: "Count Zero" To: "Vikas Gorur" Cc: gluster-users@gluster.org Sent: Thursday, May 6, 2010 1:45:32 AM Subject: Re: [Gluster-users] Replicate over WAN? > > > Replicate is not really designed for a WAN environment. A couple of things > that are probably affecting you are: > > 1) "lookup" (first access to any file or directory) needs to be sent to all > subvolumes to gather information to determine if self-heal is needed. There > is no way to fix this without losing the ability to self-heal. > > 2) "readdir" (ls) is always sent to the first subvolume. This is necessary to > ensure consistent inode numbers. Perhaps you could ensure that the first > subvolume is local? (Make sure the order of subvolumes is the same on all > your clients.) > Wait a sec. I believe there's a possible conflict in point (2), or perhaps I misunderstood: - Ensure the first subvolume is the local one - I am assuming that in order to do this, I need make the local subvolume the first one in the list of volumes, in the 'subvolumes' option - If I do this on every client, it means the order of subvolumes can not be the same, since on every client the local subvolume will be the first in the list. Can you confirm the above is ok? Thanks! :-) Also, I noticed this morning 3 patches were released to address the WAN issue? I will try them on a parallel test system first, and run some measurements. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replicate over WAN?
On May 5, 2010, at 1:15 PM, Count Zero wrote: > > Wait a sec. I believe there's a possible conflict in point (2), or perhaps I > misunderstood: > > - Ensure the first subvolume is the local one - I am assuming that in order > to do this, I need make the local subvolume the first one in the list of > volumes, in the 'subvolumes' option > - If I do this on every client, it means the order of subvolumes can not be > the same, since on every client the local subvolume will be the first in the > list. Your understanding is correct. You can set first = local on only one of your sites. -- Vikas Gorur Engineer - Gluster, Inc. -- ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replicate over WAN?
> > > Replicate is not really designed for a WAN environment. A couple of things > that are probably affecting you are: > > 1) "lookup" (first access to any file or directory) needs to be sent to all > subvolumes to gather information to determine if self-heal is needed. There > is no way to fix this without losing the ability to self-heal. > > 2) "readdir" (ls) is always sent to the first subvolume. This is necessary to > ensure consistent inode numbers. Perhaps you could ensure that the first > subvolume is local? (Make sure the order of subvolumes is the same on all > your clients.) > Wait a sec. I believe there's a possible conflict in point (2), or perhaps I misunderstood: - Ensure the first subvolume is the local one - I am assuming that in order to do this, I need make the local subvolume the first one in the list of volumes, in the 'subvolumes' option - If I do this on every client, it means the order of subvolumes can not be the same, since on every client the local subvolume will be the first in the list. Can you confirm the above is ok? Thanks! :-) Also, I noticed this morning 3 patches were released to address the WAN issue? I will try them on a parallel test system first, and run some measurements. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replicate over WAN?
On May 4, 2010, at 9:01 PM, Count Zero wrote: > I have some machines over WAN, in a Replicate cluster, with around 50ms ~ > 60ms between them. > However, I have read-subvolume specified so that it will always use the local > brick instead of the WAN. > I just need this in order to replicate files as easily and as quickly as > possible between various systems... > > My question is - Why is it still painfully slow when all I do is read > operations? > Even just listing a directory takes ages. > I can understand about Write operations, but read? and from the local > sub-volume? Replicate is not really designed for a WAN environment. A couple of things that are probably affecting you are: 1) "lookup" (first access to any file or directory) needs to be sent to all subvolumes to gather information to determine if self-heal is needed. There is no way to fix this without losing the ability to self-heal. 2) "readdir" (ls) is always sent to the first subvolume. This is necessary to ensure consistent inode numbers. Perhaps you could ensure that the first subvolume is local? (Make sure the order of subvolumes is the same on all your clients.) -- Vikas Gorur Engineer - Gluster, Inc. -- ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Replicate over WAN?
I have some machines over WAN, in a Replicate cluster, with around 50ms ~ 60ms between them. However, I have read-subvolume specified so that it will always use the local brick instead of the WAN. I just need this in order to replicate files as easily and as quickly as possible between various systems... My question is - Why is it still painfully slow when all I do is read operations? Even just listing a directory takes ages. I can understand about Write operations, but read? and from the local sub-volume? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
