Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
To close out this thread I rebuilt the entire volume with ZFS instead of ext4 and it now appears to work well. I have not loaded it up yet but the ls / ll hanging problem is gone and I can move data to and from the mounted filesystem via a glusterfs mount. Thanks, ~Mike C. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Robert Hajime Lanning Sent: Thursday, January 31, 2013 7:12 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) On 01/31/13 19:01, Robert Hajime Lanning wrote: > On 01/31/13 17:58, Michael Colonno wrote: >> Doing more searching, it appears there are several posts on various >> forums describing the same issue with Gluster deployments of 4+ nodes >> (e.g. >> http://serverfault.com/questions/453345/certain-operations-like-ls-ha >> ng-in-4 >> >> -node-gluster-setup). None of these posts are answered which doesn't >> fill me with confidence. Can anyone confirm a 4+ node cluster with >> replication that does not show this (indefinite hang on ls or ll) >> behavior? >> > > Volume Name: vol01 > Type: Distributed-Replicate > Volume ID: 3d5d1791-044e-4cf3-b17b-79c74578e590 > Status: Started > Number of Bricks: 24 x 2 = 48 > Transport-type: tcp > > 4 bricks per server * 12 servers > All bricks are 37TB logical volumes formated with XFS. > > Filesystem Size Used Avail Use% Mounted on > gfs-vol:/vol01 888T 119G 888T 1% /glusterfs/vol01 > > No issues. > Sorry, should have included: CentOS 6.3 GlusterFS 3.3.1 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
On 01/31/13 19:01, Robert Hajime Lanning wrote: On 01/31/13 17:58, Michael Colonno wrote: Doing more searching, it appears there are several posts on various forums describing the same issue with Gluster deployments of 4+ nodes (e.g. http://serverfault.com/questions/453345/certain-operations-like-ls-hang-in-4 -node-gluster-setup). None of these posts are answered which doesn't fill me with confidence. Can anyone confirm a 4+ node cluster with replication that does not show this (indefinite hang on ls or ll) behavior? Volume Name: vol01 Type: Distributed-Replicate Volume ID: 3d5d1791-044e-4cf3-b17b-79c74578e590 Status: Started Number of Bricks: 24 x 2 = 48 Transport-type: tcp 4 bricks per server * 12 servers All bricks are 37TB logical volumes formated with XFS. Filesystem Size Used Avail Use% Mounted on gfs-vol:/vol01 888T 119G 888T 1% /glusterfs/vol01 No issues. Sorry, should have included: CentOS 6.3 GlusterFS 3.3.1 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
On 01/31/13 17:58, Michael Colonno wrote: Doing more searching, it appears there are several posts on various forums describing the same issue with Gluster deployments of 4+ nodes (e.g. http://serverfault.com/questions/453345/certain-operations-like-ls-hang-in-4 -node-gluster-setup). None of these posts are answered which doesn't fill me with confidence. Can anyone confirm a 4+ node cluster with replication that does not show this (indefinite hang on ls or ll) behavior? Volume Name: vol01 Type: Distributed-Replicate Volume ID: 3d5d1791-044e-4cf3-b17b-79c74578e590 Status: Started Number of Bricks: 24 x 2 = 48 Transport-type: tcp 4 bricks per server * 12 servers All bricks are 37TB logical volumes formated with XFS. FilesystemSize Used Avail Use% Mounted on gfs-vol:/vol01888T 119G 888T 1% /glusterfs/vol01 No issues. -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
That's probably because in a 2-node replicated environment, there is no d_off transformation (at the DHT level) and therefore the bug is "avoided". Avati On Thu, Jan 31, 2013 at 5:58 PM, Michael Colonno wrote: > Doing more searching, it appears there are several posts on various > forums describing the same issue with Gluster deployments of 4+ nodes (e.g. > > http://serverfault.com/questions/453345/certain-operations-like-ls-hang-in-4 > -node-gluster-setup). None of these posts are answered which doesn't fill > me > with confidence. Can anyone confirm a 4+ node cluster with replication that > does not show this (indefinite hang on ls or ll) behavior? > > Thanks, > ~Mike C. > > -Original Message- > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno > Sent: Thursday, January 31, 2013 5:48 PM > To: gluster-users@gluster.org > Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) > > I rebuilt the volume with only TCP transport; same issues occurring > so RDMA must not be the issue here. With a glusterfs mount (which is > successful) I can copy files into the mounted volume but any command to > list > the contents (ls or ll) hangs forever while ls (for example uses ~ 5% of > the > system's CPU and glusterfs uses ~ 50% of the system's CPU). I've never seen > this behavior before with my other Gluster deployments, even much larger / > more complicated ones. This one is pretty ordinary. Any advice appreciated. > > Thanks, > ~Mike C. > > -Original Message- > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno > Sent: Thursday, January 31, 2013 4:50 PM > To: gluster-users@gluster.org > Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) > > Tried the proto=tcp setting, same error unfortunately... > > Thanks, > ~Mike C. > > -Original Message- > From: Robert Hajime Lanning [mailto:lann...@lanning.cc] > Sent: Thursday, January 31, 2013 4:28 PM > To: Michael Colonno > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) > > On 01/31/13 15:54, Michael Colonno wrote: > > Updating this thread: I tried to switch over to NFS mounting. When I > > try to mount the volume with the following line in fstab: > > > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 > > -- > Mr. Flibble > King of the Potato People > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Many thanks - I'll give this a try shortly and report results for the thread. ~Mike C. From: Anand Avati [mailto:anand.av...@gmail.com] Sent: Thursday, January 31, 2013 5:55 PM To: Michael Colonno Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) Looks like you are using a recent kernel with ext4 as the brick filesystem. It is a known issue with ext4 (http://joejulian.name/blog/glusterfs-bit-by-ext4-structure-change/). Please use XFS as the brick filesystem for now. Avati On Thu, Jan 31, 2013 at 5:47 PM, Michael Colonno wrote: I rebuilt the volume with only TCP transport; same issues occurring so RDMA must not be the issue here. With a glusterfs mount (which is successful) I can copy files into the mounted volume but any command to list the contents (ls or ll) hangs forever while ls (for example uses ~ 5% of the system's CPU and glusterfs uses ~ 50% of the system's CPU). I've never seen this behavior before with my other Gluster deployments, even much larger / more complicated ones. This one is pretty ordinary. Any advice appreciated. Thanks, ~Mike C. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno Sent: Thursday, January 31, 2013 4:50 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) Tried the proto=tcp setting, same error unfortunately... Thanks, ~Mike C. -Original Message- From: Robert Hajime Lanning [mailto:lann...@lanning.cc] Sent: Thursday, January 31, 2013 4:28 PM To: Michael Colonno Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) On 01/31/13 15:54, Michael Colonno wrote: > Updating this thread: I tried to switch over to NFS mounting. When I > try to mount the volume with the following line in fstab: > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Doing more searching, it appears there are several posts on various forums describing the same issue with Gluster deployments of 4+ nodes (e.g. http://serverfault.com/questions/453345/certain-operations-like-ls-hang-in-4 -node-gluster-setup). None of these posts are answered which doesn't fill me with confidence. Can anyone confirm a 4+ node cluster with replication that does not show this (indefinite hang on ls or ll) behavior? Thanks, ~Mike C. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno Sent: Thursday, January 31, 2013 5:48 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) I rebuilt the volume with only TCP transport; same issues occurring so RDMA must not be the issue here. With a glusterfs mount (which is successful) I can copy files into the mounted volume but any command to list the contents (ls or ll) hangs forever while ls (for example uses ~ 5% of the system's CPU and glusterfs uses ~ 50% of the system's CPU). I've never seen this behavior before with my other Gluster deployments, even much larger / more complicated ones. This one is pretty ordinary. Any advice appreciated. Thanks, ~Mike C. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno Sent: Thursday, January 31, 2013 4:50 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) Tried the proto=tcp setting, same error unfortunately... Thanks, ~Mike C. -Original Message- From: Robert Hajime Lanning [mailto:lann...@lanning.cc] Sent: Thursday, January 31, 2013 4:28 PM To: Michael Colonno Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) On 01/31/13 15:54, Michael Colonno wrote: > Updating this thread: I tried to switch over to NFS mounting. When I > try to mount the volume with the following line in fstab: > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Looks like you are using a recent kernel with ext4 as the brick filesystem. It is a known issue with ext4 ( http://joejulian.name/blog/glusterfs-bit-by-ext4-structure-change/). Please use XFS as the brick filesystem for now. Avati On Thu, Jan 31, 2013 at 5:47 PM, Michael Colonno wrote: > I rebuilt the volume with only TCP transport; same issues occurring > so RDMA must not be the issue here. With a glusterfs mount (which is > successful) I can copy files into the mounted volume but any command to > list > the contents (ls or ll) hangs forever while ls (for example uses ~ 5% of > the > system's CPU and glusterfs uses ~ 50% of the system's CPU). I've never seen > this behavior before with my other Gluster deployments, even much larger / > more complicated ones. This one is pretty ordinary. Any advice appreciated. > > Thanks, > ~Mike C. > > -Original Message- > From: gluster-users-boun...@gluster.org > [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno > Sent: Thursday, January 31, 2013 4:50 PM > To: gluster-users@gluster.org > Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) > > Tried the proto=tcp setting, same error unfortunately... > > Thanks, > ~Mike C. > > -Original Message- > From: Robert Hajime Lanning [mailto:lann...@lanning.cc] > Sent: Thursday, January 31, 2013 4:28 PM > To: Michael Colonno > Cc: gluster-users@gluster.org > Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) > > On 01/31/13 15:54, Michael Colonno wrote: > > Updating this thread: I tried to switch over to NFS mounting. When I > > try to mount the volume with the following line in fstab: > > > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 > > -- > Mr. Flibble > King of the Potato People > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
I rebuilt the volume with only TCP transport; same issues occurring so RDMA must not be the issue here. With a glusterfs mount (which is successful) I can copy files into the mounted volume but any command to list the contents (ls or ll) hangs forever while ls (for example uses ~ 5% of the system's CPU and glusterfs uses ~ 50% of the system's CPU). I've never seen this behavior before with my other Gluster deployments, even much larger / more complicated ones. This one is pretty ordinary. Any advice appreciated. Thanks, ~Mike C. -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno Sent: Thursday, January 31, 2013 4:50 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) Tried the proto=tcp setting, same error unfortunately... Thanks, ~Mike C. -Original Message- From: Robert Hajime Lanning [mailto:lann...@lanning.cc] Sent: Thursday, January 31, 2013 4:28 PM To: Michael Colonno Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) On 01/31/13 15:54, Michael Colonno wrote: > Updating this thread: I tried to switch over to NFS mounting. When I > try to mount the volume with the following line in fstab: > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On 01/31/2013 03:57 PM, Stephan von Krawczynski wrote: On Thu, 31 Jan 2013 16:00:38 -0500 Jeff Darcy wrote: Most common network errors are not a matter of design, but of dead iron. It's usually both - a design that is insufficiently tolerant of component failure, plus a combination of component failures that exceeds that tolerance. You seem to have a very high standard for filesystems continuing to maintain 100% functionality - and I suppose 100% performance as well - if there's any possibility whatsoever that they could do so. Why don't you apply that same standard to the part of the system that you're responsible for designing? Running any distributed system on top of a deficient network infrastructure will lead only to disappointment. I am sorry that glusterfs is part of the design and your critics. This sentence is incomprehensible. GlusterFS is part of the design and his/our critics? I haven't seen anything from GlusterFS criticizing anyone. Everyone working sufficiently long with networks of all kinds of sizes and components can tell you that in the end you want a design for a file service that works as long as possible. This means it should survive even if there is only one client and server and network path left. Most users, if they're down to one client and one server, they're out of business anyway. I know we can't run our whole company on one or two computers, and we're not even all that big. At least that is what is expected from glusterfs. Unfortunately sometimes you get disappointed. We saw just about everything happening when switching off all but one reliable network path including network hangs and server hangs (the last one) (read the list for examples by others). After taking a power hit to the building which was able to get through our commercial ups and line conditioners (which also had to be replaced after that), our switches all had to be replaced. Part of my testing was to do just that. I pulled the plugs on all the switches, leaving only one data path then changing that path. All the while I had full activity on all my volumes, including innodb transactions, vm image activity, web, samba, and workstation home directories. I ran multiple dd tests from multiple clients during these tests. Not once did they even slow down. On the other end of the story clients see servers go offline if you increase the non-gluster traffic on the network. Main (but not only) reason is the very low default ping time (read the list for examples by others). 42 seconds is low? I've never (and I used to have really crappy dual 32bit xeon servers) saturated my systems to the point where it took 42 seconds for a server to respond. If you're switches can't handle that kind of traffic, perhaps you're using the wrong hardware. All these seen effects show clearly that noone ever tested this to an extent I would have done writing this kind of software. After all this is a piece of software whose merely only purpose is surviving dead servers and networks. It is no question of design, because on paper everything looks promising. Sometimes your arguments let me believe you want glusterfs working like a ford car. A lot of technical gameplay built in but the idea that a car should be a good car in the first place got lost on the way somewhere. Quite a lot of the features built in lately have the quality of an mp3-player in your ford. Nice to have but does not help you a lot driving 200 and a rabbit crossing. And this is why I am requesting the equivalent of a BMW. Using your own analogy, what good is a BMW if you've got no roads to drive it on? You're talking in terms of single entities. Most (if not all) of the sysadmins I work with on a daily basis, my peers in the industry, members of LOPSA... we work in systems. We know how to build in redundancy and plan for and survive failures. There's not a week that goes by where something in my system hasn't encountered some issue, yet our company has not lost any productivity because of it. We have the best availability of systems of all our competitors (and I do monitor their systems). I'm not saying you're doing it wrong. Be as asssured as you feel is appropriate for your system requirements. Just be aware that the majority of the industry does not share those requirements. I'm not speaking from my "gut instinct" but I am a member of several professional organizations. I attend functions on a weekly basis attended by members of organizations like Expedia, Ebay, Amazon, Google, Boeing, Starbucks, etc, etc. and we talk about this stuff. Fault tolerance and recovery is a big part of what we do, probably the biggest, and I still advise the way I do, not through just my own experiences, but through the experiences of my peers. Offer advice and back it up with facts, anecdotes, and/or tests, but accept that there are as many ways of managing systems as there are systems to be managed. Accept that there are professionals in th
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Tried the proto=tcp setting, same error unfortunately... Thanks, ~Mike C. -Original Message- From: Robert Hajime Lanning [mailto:lann...@lanning.cc] Sent: Thursday, January 31, 2013 4:28 PM To: Michael Colonno Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) On 01/31/13 15:54, Michael Colonno wrote: > Updating this thread: I tried to switch over to NFS mounting. When I > try to mount the volume with the following line in fstab: > > node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
On 01/31/2013 04:00 PM, Michael Colonno wrote: Related: any official word on full RDMA support in Gluster 3.x? Thanks, ~Mike C. The last official word I got from Raghavendra Bhat defined the state of RDMA in 3.3.1 as "Tech Preview". As I understand it, this is a Red Hat term for "it hasn't gone through sufficient QA testing for us to support it commercially". There's no less support in 3.3 than there was in 3.1 or 3.2. The people that I've seen use RDMA successfully used bleeding edge kernels and drivers, fwiw. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
On 01/31/13 15:54, Michael Colonno wrote: Updating this thread: I tried to switch over to NFS mounting. When I try to mount the volume with the following line in fstab: node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3 0 0 node1:/Volume /tmp/mnt nfs defaults,_netdev,vers=3,proto=tcp 0 0 -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Do I need to blow up and rebuild the brick to make that happen or can this be set on the fly? Possibly relevant fact: I do not have my IB fabric in place yet but I'm happy to use IPoIB for this deployment when I do. I included it as a placeholder. Related: any official word on full RDMA support in Gluster 3.x? Thanks, ~Mike C. From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Bryan Whitehead Sent: Thursday, January 31, 2013 3:52 PM To: Michael Colonno Cc: gluster-users Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) remove the transport rdma and try again. When using RDMA I've also had extremely bad CPU eating issues. I currently run gluster with IPoIB to get the speed of infiniband and the non-crazy cpu usage of rdma gluster. On Thu, Jan 31, 2013 at 9:20 AM, Michael Colonno wrote: Hi All ~ I created an eight-brick gluster 3.3.1 volume (2x replication) on eight CentOS 6.3 x64 systems. I was able to form and start the volume without issue. I was also able to mount it through /etc/fstab as a native glusterfs mount. I have a couple questions issues at this point: - On a client machine, "glusterfs" is not recognized as a valid type unless gluster-server is installed. This seems to contradict the documentation - wanted to make sure I'm not doing something wrong. More a clarification than issue here. - The glusterfs process is taking between 50% and 80% CPU on both the brick and client systems (these are fairly powerful, brand new servers). - No doubt linked to the above, the mounted filesystem hangs indefinitely when accessed. I tried an "ls -a" on the mounted filesystem, for example, which hangs forever. I tested this by mounting a brick system to itself and to a client which is not a brick and the same behavior was observed. Both were glusterfs mounts. There is nothing special about my deployment except for the use of transport = tcp,rdma. I am running on Ethernet now but will be migrating to Infiniband after this is debugged. Thanks for any advice, ~Mike C. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On Thu, 31 Jan 2013 16:00:38 -0500 Jeff Darcy wrote: > > Most common network errors are not a matter of design, but of dead > > iron. > > It's usually both - a design that is insufficiently tolerant of > component failure, plus a combination of component failures that exceeds > that tolerance. You seem to have a very high standard for filesystems > continuing to maintain 100% functionality - and I suppose 100% > performance as well - if there's any possibility whatsoever that they > could do so. Why don't you apply that same standard to the part of the > system that you're responsible for designing? Running any distributed > system on top of a deficient network infrastructure will lead only to > disappointment. I am sorry that glusterfs is part of the design and your critics. Everyone working sufficiently long with networks of all kinds of sizes and components can tell you that in the end you want a design for a file service that works as long as possible. This means it should survive even if there is only one client and server and network path left. At least that is what is expected from glusterfs. Unfortunately sometimes you get disappointed. We saw just about everything happening when switching off all but one reliable network path including network hangs and server hangs (the last one) (read the list for examples by others). On the other end of the story clients see servers go offline if you increase the non-gluster traffic on the network. Main (but not only) reason is the very low default ping time (read the list for examples by others). All these seen effects show clearly that noone ever tested this to an extent I would have done writing this kind of software. After all this is a piece of software whose merely only purpose is surviving dead servers and networks. It is no question of design, because on paper everything looks promising. Sometimes your arguments let me believe you want glusterfs working like a ford car. A lot of technical gameplay built in but the idea that a car should be a good car in the first place got lost on the way somewhere. Quite a lot of the features built in lately have the quality of an mp3-player in your ford. Nice to have but does not help you a lot driving 200 and a rabbit crossing. And this is why I am requesting the equivalent of a BMW. -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Updating this thread: I tried to switch over to NFS mounting. When I try to mount the volume with the following line in fstab: node1:/Volume /tmp/mntnfs defaults,_netdev,vers=3 0 0 I get: mount.nfs: mounting node1:/Volume failed, reason given by server: No such file or directory It seems from the documentation that the mount folder is supposed to be the gluster volume name. I also assume I don't need an exports entry since there is nothing in the documentation about this and the volume options seem to indicate that My gluster volume itself seems to start and stop without any issue but can't seem to mount via any method(?) Hopefully I am just doing something simple wrong in config; I have tried to meticulously follow the admin guide. For reference here is my volume info: Volume Name: Volume Type: Distributed-Replicate Volume ID: 409d4096-feda-4b5f-b01b-3383d39af1c6 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp,rdma Bricks: Brick1: node1:/volume Brick2: node2:/volume Brick3: node3:/volume Brick4: node4:/volume Brick5: node5:/volume Brick6: node6:/volume Brick7: node7:/volume Brick8: node8:/volume Thanks, ~Mike C. From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Michael Colonno Sent: Thursday, January 31, 2013 2:41 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] hanging of mounted filesystem (3.3.1) Hi All ~ I created an eight-brick gluster 3.3.1 volume (2x replication) on eight CentOS 6.3 x64 systems. I was able to form and start the volume without issue. I was also able to mount it through /etc/fstab as a native glusterfs mount. I have a couple questions issues at this point: - On a client machine, "glusterfs" is not recognized as a valid type unless gluster-server is installed. This seems to contradict the documentation - wanted to make sure I'm not doing something wrong. More a clarification than issue here. - The glusterfs process is taking between 50% and 80% CPU on both the brick and client systems (these are fairly powerful new servers). - No doubt linked to the above, the mounted filesystem hangs indefinitely when accessed. I tried an "ls -a" on the mounted filesystem, for example, which hangs forever. I tested this by mounting a brick system to itself and to a client which is not a brick and the same behavior was observed. Both were glusterfs mounts. There is nothing special about my deployment except for the use of transport = tcp,rdma. I am running on Ethernet now but will be migrating to Infiniband after this is debugged. Thanks for any advice, ~Mike C. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
remove the transport rdma and try again. When using RDMA I've also had extremely bad CPU eating issues. I currently run gluster with IPoIB to get the speed of infiniband and the non-crazy cpu usage of rdma gluster. On Thu, Jan 31, 2013 at 9:20 AM, Michael Colonno wrote: > Hi All ~ > > I created an eight-brick gluster 3.3.1 volume (2x replication) on > eight CentOS 6.3 x64 systems. I was able to form and start the volume > without issue. I was also able to mount it through /etc/fstab as a native > glusterfs mount. I have a couple questions issues at this point: > > - On a client machine, "glusterfs" is not recognized as a valid > type > unless gluster-server is installed. This seems to contradict the > documentation - wanted to make sure I'm not doing something wrong. More a > clarification than issue here. > > - The glusterfs process is taking between 50% and 80% CPU on both > the brick and client systems (these are fairly powerful, brand new > servers). > > > - No doubt linked to the above, the mounted filesystem hangs > indefinitely when accessed. I tried an "ls -a" on the mounted filesystem, > for example, which hangs forever. I tested this by mounting a brick system > to itself and to a client which is not a brick and the same behavior was > observed. Both were glusterfs mounts. > > There is nothing special about my deployment except for the use of > transport = tcp,rdma. I am running on Ethernet now but will be migrating to > Infiniband after this is debugged. > > Thanks for any advice, > ~Mike C. > > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Striped Replicated Volumes: create files error.
Hi there, each time I copy (or dd or similar) a file to a striped replicated volume I get an error: the argument is not valid. An empty file is created. If I now run the copy, it works. This is in independed of the client platform. We are using version 3.3.1 Mit freundlichen Grüßen / Kind regards Axel Weber ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] hanging of mounted filesystem (3.3.1)
Hi All ~ I created an eight-brick gluster 3.3.1 volume (2x replication) on eight CentOS 6.3 x64 systems. I was able to form and start the volume without issue. I was also able to mount it through /etc/fstab as a native glusterfs mount. I have a couple questions issues at this point: - On a client machine, "glusterfs" is not recognized as a valid type unless gluster-server is installed. This seems to contradict the documentation - wanted to make sure I'm not doing something wrong. More a clarification than issue here. - The glusterfs process is taking between 50% and 80% CPU on both the brick and client systems (these are fairly powerful new servers). - No doubt linked to the above, the mounted filesystem hangs indefinitely when accessed. I tried an "ls -a" on the mounted filesystem, for example, which hangs forever. I tested this by mounting a brick system to itself and to a client which is not a brick and the same behavior was observed. Both were glusterfs mounts. There is nothing special about my deployment except for the use of transport = tcp,rdma. I am running on Ethernet now but will be migrating to Infiniband after this is debugged. Thanks for any advice, ~Mike C. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] hanging of mounted filesystem (3.3.1)
Hi All ~ I created an eight-brick gluster 3.3.1 volume (2x replication) on eight CentOS 6.3 x64 systems. I was able to form and start the volume without issue. I was also able to mount it through /etc/fstab as a native glusterfs mount. I have a couple questions issues at this point: - On a client machine, "glusterfs" is not recognized as a valid type unless gluster-server is installed. This seems to contradict the documentation - wanted to make sure I'm not doing something wrong. More a clarification than issue here. - The glusterfs process is taking between 50% and 80% CPU on both the brick and client systems (these are fairly powerful new servers). - No doubt linked to the above, the mounted filesystem hangs indefinitely when accessed. I tried an "ls -a" on the mounted filesystem, for example, which hangs forever. I tested this by mounting a brick system to itself and to a client which is not a brick and the same behavior was observed. Both were glusterfs mounts. There is nothing special about my deployment except for the use of transport = tcp,rdma. I am running on Ethernet now but will be migrating to Infiniband after this is debugged. Thanks for any advice, ~Mike C. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] New version of UFO - is there a new HOWTO?
On 1/31/2013 12:36 PM, Kaleb Keithley wrote:
Not sure if you saw this in #gluster on IRC.
The other work-around for F18 is to delete
/etc/swift/{account,container,object}-server.conf before starting UFO.
With that my UFO set-up works as it did in F17.
Still no joy.
http://fpaste.org/SOYM/
Same thing happens if I remove all the packages, get rid of /etc/swift,
and reinstall. I can confirm that the file referenced in the last part
of the paste (/etc/swift/object.ring.gz) did not exist at any point. A
previous installation of 3.3.1-2 on F17 has both .builder and .ring.gz
files.
http://fpaste.org/jIm5/
Thanks,
Shawn
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On 01/31/2013 02:57 PM, Stephan von Krawczynski wrote: You are asking in the wrong direction. The simple question is: is there any dynamic configuration equally safe than a local config file? Yes, because inconsistent configurations are a danger too. There is no probability, either you are a dead client or a working one. That's a total non sequitur. As soon as there's more than one state, there are probabilities of being in each one. As it turns out, there is also a third possibility - that you are a malfunctioning or incorrectly configured client. Fail-stop errors are the easy ones. It wouldn't be useful to release a network filesystem that drops dead in case of network errors. Every network filesystem, or other distributed system of any kind, will fail if there are *sufficient* errors. Some are better than others at making that number larger, some are better than others with respect to the consequence of exceeding that threshold, but all will fail eventually. GlusterFS does handle a great many kinds of failure that you probably can't even imagine, but neither it nor any other system can handle all possible failures. Most common network errors are not a matter of design, but of dead iron. It's usually both - a design that is insufficiently tolerant of component failure, plus a combination of component failures that exceeds that tolerance. You seem to have a very high standard for filesystems continuing to maintain 100% functionality - and I suppose 100% performance as well - if there's any possibility whatsoever that they could do so. Why don't you apply that same standard to the part of the system that you're responsible for designing? Running any distributed system on top of a deficient network infrastructure will lead only to disappointment. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On Thu, 31 Jan 2013 14:17:32 -0500 Jeff Darcy wrote: > There is *always* at least one situation, however unlikely, where you're > busted. Designing reliable systems is always about probabilities. If > none of the solutions mentioned so far suffice for you, there are still > others that don't involve sacrificing the advantages of dynamic > configuration. If your network is so FUBAR that you have trouble > reaching any server to fetch a volfile, then it probably wouldn't do you > any good to have one locally because you wouldn't be able to reach those > servers for I/O anyway. You'd be just asking for split brain and other > problems. Redesigning the mount is likely to yield less benefit than > redesigning the network that's susceptible to such failures. You are asking in the wrong direction. The simple question is: is there any dynamic configuration equally safe than a local config file? If your local fs is dead, then you are really dead. But if it is alive you have a config. And that's about it. You need no working DNS, no poisoned cache and no special server that must not fail. Everything with less security is inacceptable. There is no probability, either you are a dead client or a working one. And if you really want to include the network as a question. I would expect the gluster client-server and server-server protocols to accept network failure as a default case. It wouldn't be useful to release a network filesystem that drops dead in case of network errors. If there is some chance to survive it should be able to do so and still work. Most common network errors are not a matter of design, but of dead iron. -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] New version of UFO - is there a new HOWTO?
Not sure if you saw this in #gluster on IRC.
The other work-around for F18 is to delete
/etc/swift/{account,container,object}-server.conf before starting UFO.
With that my UFO set-up works as it did in F17.
- Original Message -
From: "Kaleb S. KEITHLEY"
On 01/29/2013 11:02 PM, Shawn Heisey wrote:
> This is in the proxy-server.conf file:
>
> [pipeline:main]
> pipeline = healthcheck cache authtoken keystone proxy-server
proxy-server.conf-gluster has tempauth. That comes from the gluster-ufo
rpm. Keystone auth is coming in a future release. Soon hopefully
And yes, the conflict between glusterfs-ufo and glusterfs-swift-* is a
bug. You can work around it with:
% yumdownloader glusterfs-ufo
% rpm -i --force glustefs-ufo-3.3.1-8.fc18.noarch.rpm
>
> One thing to note: I was not able to install glusterfs-ufo - it has
> config file conflicts with the other packages - one of which
> (glusterfs-swift) it actually depends on. Does the -ufo package have
> the config file with tempauth?
>
> Thanks,
> Shawn
>
> On 1/29/2013 8:43 PM, Kaleb Keithley wrote:
>> 3.3.1-8 in f18 still uses tempauth.
>>
>> - Original Message -
>> From: "Shawn Heisey"
>> To: gluster-users@gluster.org
>> Sent: Tuesday, January 29, 2013 8:21:11 PM
>> Subject: [Gluster-users] New version of UFO - is there a new HOWTO?
>>
>> I just installed glusterfs-swift 3.3.1 on a couple of Fedora 18 servers.
>>This is based on swift 1.7.4 and has keystone in the config. I had
>> experimented with the one based on swift 1.4.8 and tempauth and had some
>> problems with it. The HOWTO I can find is still for the old one. Is
>> there an updated one?
>>
>> I would also need to find some instructions on setting up keystone from
>> scratch for use with gluster-swift.
>>
>> Thanks,
>> Shawn
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
Kaleb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On 01/31/2013 01:59 PM, Stephan von Krawczynski wrote: You don't want to use DNS in an environment where security is your first rule. If your DNS drops dead your setup is dead. Not very promising ... The basic goal of glusterfs has been to secure data by replicating it. Data distribution is really not interesting for us. Now you say "go and replicate your data for security, but use DNS to secure your setup". ??? You really seem to like Domino-setups. DNS dead => everything dead. This vulnerability can be reduced by using multiple DNS servers, and even further by using local caches/proxies which can still do round-robin for you. There have also been solutions mentioned that don't rely on DNS, so talking about DNS unreliability is a total red herring. * The mount option backupvolfile-server. An fstab entry like "server1:myvol /mnt/myvol glusterfs backupvolfile-server=server2 0 0" will allow the mount command to try server2 if server1 does not mount successfully. And how many backup servers do you want to name in your fstab? In fact you have to name all your servers because else there will always be at least one situation you are busted. There is *always* at least one situation, however unlikely, where you're busted. Designing reliable systems is always about probabilities. If none of the solutions mentioned so far suffice for you, there are still others that don't involve sacrificing the advantages of dynamic configuration. If your network is so FUBAR that you have trouble reaching any server to fetch a volfile, then it probably wouldn't do you any good to have one locally because you wouldn't be able to reach those servers for I/O anyway. You'd be just asking for split brain and other problems. Redesigning the mount is likely to yield less benefit than redesigning the network that's susceptible to such failures. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On Thu, 31 Jan 2013 09:07:50 -0800 Joe Julian wrote: > On 01/31/2013 08:38 AM, Stephan von Krawczynski wrote: > > On Thu, 31 Jan 2013 12:47:30 + > > Brian Candler wrote: > > > >> On Thu, Jan 31, 2013 at 09:18:26AM +0100, Stephan von Krawczynski wrote: > The client will still fail (in most cases) since host1 (if I follow you) > is > part of the gluster groupset. Certainly if it's a distributed-only, > maybe not > if it's a dist/repl gluster. But if host1 goes down, the client will > not be > able to find a gluster vol to mount. > >>> For sure it will not fail if replication is used. > >> Aside: it will *fail* if the client reboots, and /etc/fstab has > >> server1:/volname, and server1 is the one which failed. > > Well, this is exactly the reason we generally deny to fetch the volfile from > > the server. This whole idea is obvious nonsense for exactly the reason you > > described. > > > That doesn't lend me much confidence in your expertise with regard to > your other recommendations, Stephan. > > There are two good ways to make this work even if a server is down: > > * Round robin DNS. A hostname (ie. glusterfs.domain.dom) with multiple > A records that point to all your servers. Using that hostname in > fstab will allow the client to roll over to the additional servers > in the event the first one it gets is not available (ie. > "glusterfs.domain.dom:myvol /mnt/myvol glusterfs defaults 0 0"). You don't want to use DNS in an environment where security is your first rule. If your DNS drops dead your setup is dead. Not very promising ... The basic goal of glusterfs has been to secure data by replicating it. Data distribution is really not interesting for us. Now you say "go and replicate your data for security, but use DNS to secure your setup". ??? You really seem to like Domino-setups. DNS dead => everything dead. > * The mount option backupvolfile-server. An fstab entry like > "server1:myvol /mnt/myvol glusterfs backupvolfile-server=server2 0 > 0" will allow the mount command to try server2 if server1 does not > mount successfully. And how many backup servers do you want to name in your fstab? In fact you have to name all your servers because else there will always be at least one situation you are busted. > This whole idea is obvious experience and forethought, not nonsense. By > having a management service that provides configuration, on-the-fly > configuration changes are possible. If one "denies to fetch the volfile" > one cripples their cluster's flexibility. I don't know what kind of setups you drive. In our environment we don't want to fiddle around with fs configs. We want them to work as expected even if other parts of the total setup fall apart. Flexibility in our world means you can do widespread types of configurations. It does not mean we switch the running configs every day only because gluster is so flexible. -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Extremely high CPU load on one server
On 01/31/2013 05:37 PM, gluster-users-requ...@gluster.org wrote: Date: Thu, 31 Jan 2013 15:28:25 + (UTC) From: Adri? Garc?a-Alz?rriz To: gluster-users@gluster.org Subject: Re: [Gluster-users] Extremely high CPU load on one server Message-ID: Content-Type: text/plain; charset="us-ascii" El dia Thu, 31 Jan 2013 15:02:25 +, en/na Dan Bretherton va escriure: Can anyone suggest a way to troubleshoot this problem? The rebalance logs don't show anything unusual but glustershd.log has a lot of metadata split-brain warnings. The brick logs are full of scary looking warnings but none flagged 'E' or 'C'. The trouble is that I see messages like these on all the servers, and I can find nothing unusual about the server with a CPU load of 70. Users are complaining about very poor performance, which has been going on or several weeks, so I must at least find a work-around that allows people to work normally. -Dan. What's the output for your gluster volume info command? What kind of disks are you using for? How is the wait parameter in top? Hello- Here is the information you requested; thanks. What's the output for your gluster volume info command? There are ten volumes in the cluster, and the one I'm having the most trouble with is shown below. [root@bdan11 ~]# gluster volume info atmos Volume Name: atmos Type: Distributed-Replicate Volume ID: a4a7774e-cf4d-4a3e-9477-5c3d1b0efd93 Status: Started Number of Bricks: 16 x 2 = 32 Transport-type: tcp Bricks: Brick1: bdan0.nerc-essc.ac.uk:/local/glusterfs Brick2: bdan1.nerc-essc.ac.uk:/local/glusterfs Brick3: bdan2.nerc-essc.ac.uk:/local/glusterfs Brick4: bdan3.nerc-essc.ac.uk:/local/glusterfs Brick5: bdan4.nerc-essc.ac.uk:/atmos/glusterfs Brick6: bdan5.nerc-essc.ac.uk:/atmos/glusterfs Brick7: bdan6.nerc-essc.ac.uk:/local/glusterfs Brick8: bdan7.nerc-essc.ac.uk:/local/glusterfs Brick9: bdan8.nerc-essc.ac.uk:/atmos/glusterfs Brick10: bdan9.nerc-essc.ac.uk:/atmos/glusterfs Brick11: bdan10.nerc-essc.ac.uk:/atmos/glusterfs Brick12: bdan11.nerc-essc.ac.uk:/atmos/glusterfs Brick13: bdan12.nerc-essc.ac.uk:/atmos/glusterfs Brick14: bdan13.nerc-essc.ac.uk:/atmos/glusterfs Brick15: bdan10.nerc-essc.ac.uk:/atmos2/glusterfs Brick16: bdan11.nerc-essc.ac.uk:/atmos2/glusterfs Brick17: bdan12.nerc-essc.ac.uk:/atmos2/glusterfs Brick18: bdan13.nerc-essc.ac.uk:/atmos2/glusterfs Brick19: bdan4.nerc-essc.ac.uk:/atmos2/glusterfs Brick20: bdan5.nerc-essc.ac.uk:/atmos2/glusterfs Brick21: bdan12.nerc-essc.ac.uk:/atmos3/glusterfs Brick22: bdan13.nerc-essc.ac.uk:/atmos3/glusterfs Brick23: bdan12.nerc-essc.ac.uk:/atmos4/glusterfs Brick24: bdan13.nerc-essc.ac.uk:/atmos4/glusterfs Brick25: bdan12.nerc-essc.ac.uk:/atmos5/glusterfs Brick26: bdan13.nerc-essc.ac.uk:/atmos5/glusterfs Brick27: pegasus.nerc-essc.ac.uk:/atmos/glusterfs Brick28: bdan14.nerc-essc.ac.uk:/atmos/glusterfs Brick29: bdan15.nerc-essc.ac.uk:/atmos/glusterfs Brick30: bdan16.nerc-essc.ac.uk:/atmos/glusterfs Brick31: bdan15.nerc-essc.ac.uk:/atmos2/glusterfs Brick32: bdan16.nerc-essc.ac.uk:/atmos2/glusterfs Options Reconfigured: nfs.enable-ino32: on server.allow-insecure: on performance.stat-prefetch: off performance.quick-read: off cluster.min-free-disk: 338GB nfs.rpc-auth-allow: 192.171.166.*,134.225.100.* features.quota: off What kind of disks are you using for? They are Hitachi Deskstar 7200 RPM 2TB SATA drives, attached to an Areca ARC1880 RAID controller. I'm aware that it's an unusual choice of drive but the server vendor swears by them. I have another pair of servers from the same vendor with 1TB Hitachi Deskstars and they have not given any GlusterFS related trouble. The bricks are logical volumes on top of RAID-6 provided by the Areca. On this particular server they are formatted ext4, but I recently started using XFS for more recently added bricks on newer servers. The bricks vary in size from 3.3TB to 500GB, depending on which volume they belong to. I use smaller bricks for smaller volumes so that the brick size is an appropriate increment of volume growth. How is the wait parameter in top? There isn't much I/O wait as far as I can gather. Here is a view showing the top few processes. top - 18:22:58 up 1:59, 1 user, load average: 64.25, 64.36, 64.14 Tasks: 218 total, 9 running, 209 sleeping, 0 stopped, 0 zombie Cpu(s): 78.2%us, 20.7%sy, 0.0%ni, 0.9%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 12290244k total, 6639128k used, 5651116k free, 2752944k buffers Swap: 14352376k total,0k used, 14352376k free, 1441712k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 3661 root 15 0 289m 42m 2168 R 85.0 0.4 101:54.33 glusterfsd 3697 root 15 0 304m 37m 2112 S 83.1 0.3 75:53.43 glusterfsd 3655 root 15 0 289m 42m 2160 S 79.4 0.4 97:11.52 glusterfsd 3685 root 15 0 550m 62m 2100 S 77.5 0.5 109:07.53 glusterfsd 3691 root 15 0 483m 40m 2100 S 75.6 0.3 76:10.93 glusterfsd 3679 root
Re: [Gluster-users] NFS availability
On Thu, 2013-01-31 at 09:07 -0800, Joe Julian wrote: > That doesn't lend me much confidence in your expertise with regard to > your other recommendations, Stephan. > > There are two good ways to make this work even if a server is down: I'd just like to add one more option to the already excellent list that Joe has provided. It happens to be my favourite: VRRP. You can use something like Keepalived or even cman+rgmanager+an ip resource, to manager a "virtual IP". Each of your servers has an IP, and then this extra VIP moves around automatically based on what is actually up, and where you want the mount to come from. If you want to be particularly fastidious, then I suppose you could even create a second VIP (which defaults to a different "base" node, and which you use the below "backupvolfile-server" option. VRRP has the added benefit that the failover happens within seconds, and can be set manually if for some reason you want to choose which volfile server you have. HTH, James > > * Round robin DNS. A hostname (ie. glusterfs.domain.dom) with > multiple > A records that point to all your servers. Using that hostname in > fstab will allow the client to roll over to the additional servers > in the event the first one it gets is not available (ie. > "glusterfs.domain.dom:myvol /mnt/myvol glusterfs defaults 0 0"). > * The mount option backupvolfile-server. An fstab entry like > "server1:myvol /mnt/myvol glusterfs backupvolfile-server=server2 0 > 0" will allow the mount command to try server2 if server1 does not > mount successfully. > > This whole idea is obvious experience and forethought, not nonsense. > By > having a management service that provides configuration, on-the-fly > configuration changes are possible. If one "denies to fetch the > volfile" > one cripples their cluster's flexibility. signature.asc Description: This is a digitally signed message part ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] hanging of mounted filesystem (3.3.1)
Hi All ~ I created an eight-brick gluster 3.3.1 volume (2x replication) on eight CentOS 6.3 x64 systems. I was able to form and start the volume without issue. I was also able to mount it through /etc/fstab as a native glusterfs mount. I have a couple questions issues at this point: - On a client machine, "glusterfs" is not recognized as a valid type unless gluster-server is installed. This seems to contradict the documentation - wanted to make sure I'm not doing something wrong. More a clarification than issue here. - The glusterfs process is taking between 50% and 80% CPU on both the brick and client systems (these are fairly powerful, brand new servers). - No doubt linked to the above, the mounted filesystem hangs indefinitely when accessed. I tried an "ls -a" on the mounted filesystem, for example, which hangs forever. I tested this by mounting a brick system to itself and to a client which is not a brick and the same behavior was observed. Both were glusterfs mounts. There is nothing special about my deployment except for the use of transport = tcp,rdma. I am running on Ethernet now but will be migrating to Infiniband after this is debugged. Thanks for any advice, ~Mike C. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On 01/31/2013 08:38 AM, Stephan von Krawczynski wrote: On Thu, 31 Jan 2013 12:47:30 + Brian Candler wrote: On Thu, Jan 31, 2013 at 09:18:26AM +0100, Stephan von Krawczynski wrote: The client will still fail (in most cases) since host1 (if I follow you) is part of the gluster groupset. Certainly if it's a distributed-only, maybe not if it's a dist/repl gluster. But if host1 goes down, the client will not be able to find a gluster vol to mount. For sure it will not fail if replication is used. Aside: it will *fail* if the client reboots, and /etc/fstab has server1:/volname, and server1 is the one which failed. Well, this is exactly the reason we generally deny to fetch the volfile from the server. This whole idea is obvious nonsense for exactly the reason you described. That doesn't lend me much confidence in your expertise with regard to your other recommendations, Stephan. There are two good ways to make this work even if a server is down: * Round robin DNS. A hostname (ie. glusterfs.domain.dom) with multiple A records that point to all your servers. Using that hostname in fstab will allow the client to roll over to the additional servers in the event the first one it gets is not available (ie. "glusterfs.domain.dom:myvol /mnt/myvol glusterfs defaults 0 0"). * The mount option backupvolfile-server. An fstab entry like "server1:myvol /mnt/myvol glusterfs backupvolfile-server=server2 0 0" will allow the mount command to try server2 if server1 does not mount successfully. This whole idea is obvious experience and forethought, not nonsense. By having a management service that provides configuration, on-the-fly configuration changes are possible. If one "denies to fetch the volfile" one cripples their cluster's flexibility. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On Thu, 31 Jan 2013 12:47:30 + Brian Candler wrote: > On Thu, Jan 31, 2013 at 09:18:26AM +0100, Stephan von Krawczynski wrote: > > > The client will still fail (in most cases) since host1 (if I follow you) > > > is > > > part of the gluster groupset. Certainly if it's a distributed-only, maybe > > > not > > > if it's a dist/repl gluster. But if host1 goes down, the client will not > > > be > > > able to find a gluster vol to mount. > > > > For sure it will not fail if replication is used. > > Aside: it will *fail* if the client reboots, and /etc/fstab has > server1:/volname, and server1 is the one which failed. Well, this is exactly the reason we generally deny to fetch the volfile from the server. This whole idea is obvious nonsense for exactly the reason you described. -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Extremely high CPU load on one server
El dia Thu, 31 Jan 2013 15:02:25 +, en/na Dan Bretherton va escriure: > Can anyone suggest a way to troubleshoot this problem? The rebalance > logs don't show anything unusual but glustershd.log has a lot of > metadata split-brain warnings. The brick logs are full of scary > looking warnings but none flagged 'E' or 'C'. The trouble is that I see > messages like these on all the servers, and I can find nothing unusual > about the server with a CPU load of 70. Users are complaining about > very poor performance, which has been going on or several weeks, so I > must at least find a work-around that allows people to work normally. > > -Dan. What's the output for your gluster volume info command? What kind of disks are you using for? How is the wait parameter in top? signature.asc Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Extremely high CPU load on one server
Dear All- I originally asked this question in another thread, but as it is a separate problem I thought it deserved its own thread. I had to extend two volumes recently and the layout fixes for both of them are now running at the same time. One server now has a load of over 70 most of the time (mostly glusterfsd), but none of the others seem to be particularly busy. I restarted the server in question but the CPU load quickly went up to 70 again. I can't see any particular reason why this one server should be so badly affected by the layout fixing processes. It isn't a particularly big server, with only five 3TB bricks involved in the two volumes that were extended. One possibility is that a lot of batch jobs on our compute cluster are accessing the same set of files that just happen to be on this one server, which could potentially happen because I have not been able to rebalance the files in the storage volumes for a long time after the last attempt resulted in data corruption (in GlusterFS version 3.2). However, poorly distributed files certainly isn't the whole story, because even when there is not much running on the compute cluster the storage server load remains extremely high. Can anyone suggest a way to troubleshoot this problem? The rebalance logs don't show anything unusual but glustershd.log has a lot of metadata split-brain warnings. The brick logs are full of scary looking warnings but none flagged 'E' or 'C'. The trouble is that I see messages like these on all the servers, and I can find nothing unusual about the server with a CPU load of 70. Users are complaining about very poor performance, which has been going on or several weeks, so I must at least find a work-around that allows people to work normally. -Dan. -- Mr. D.A. Bretherton Computer System Manager Environmental Systems Science Centre (ESSC) Department of Meteorology Harry Pitt Building 3 Earley Gate University of Reading Reading, RG6 7BE (or RG6 6AL for postal service deliveries) UK Tel. +44 118 378 5205, Fax: +44 118 378 6413 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On Thu, Jan 31, 2013 at 09:18:26AM +0100, Stephan von Krawczynski wrote: > > The client will still fail (in most cases) since host1 (if I follow you) is > > part of the gluster groupset. Certainly if it's a distributed-only, maybe > > not > > if it's a dist/repl gluster. But if host1 goes down, the client will not > > be > > able to find a gluster vol to mount. > > For sure it will not fail if replication is used. Aside: it will *fail* if the client reboots, and /etc/fstab has server1:/volname, and server1 is the one which failed. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS availability
On Wed, 30 Jan 2013 20:44:52 -0800 harry mangalam wrote: > On Thursday, January 31, 2013 11:28:04 AM glusterzhxue wrote: > > Hi all, > > As is known to us all, gluster provides NFS mount. However, if the mount > > point fails, clients will lose connection to Gluster. While if we use > > gluster native client, this fail will have no effect on clients. For > > example: > mount -t glusterfs host1:/vol1 /mnt > > > > If host1 goes down for some reason, client works still, it has no sense > > about the failure(suppose we have multiple gluster servers). > > The client will still fail (in most cases) since host1 (if I follow you) is > part of the gluster groupset. Certainly if it's a distributed-only, maybe not > if it's a dist/repl gluster. But if host1 goes down, the client will not be > able to find a gluster vol to mount. For sure it will not fail if replication is used. > > However, if > > we use the following: > mount -t nfs -o vers=3 host1:/vol1 /mnt > > > If host1 failed, client will lose connection to gluster servers. > > If the client was mounting the glusterfs via a re-export from an intermediate > host, you might be able to failover to another intermediate NFS server, but > if > it was a gluster host, it would fail due to the reasons above. kernel-nfs _may_ failover from server A to server B if B takes the original server IP and some requirements are met. You don't need an intermediate (re-exporting) server for this. -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
