Re: [Gluster-users] MTU size issue?
Hi Atin, You’re right in saying if it’s activate then all nodes should have it activated. What I find strange is that when glusterfsd has problems communicating with the other peers that that single node with issues isn’t considered “not connected” and thus expelled from the cluster somehow; in my case it caused a complete hang of the trusted storage pool. And to emphasise this, pinging was no problem as it uses small packets anyway so jumbo frames were not used at all… enabling jumbo frames on the interface and switches is only a way to tell the TCP/IP stack that it can send larger packets but it does’t have to. Or am I mistaking in that the TCP/IP stack will control wether to send the bigger packets and that glusterfsd has no control over that? Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl | www.surfsara.nl | Regular day off on friday > On 15 Oct 2015, at 08:24, Atin Mukherjee <amukh...@redhat.com> wrote: > > > > On 10/14/2015 05:09 PM, Sander Zijlstra wrote: >> LS, >> >> I recently reconfigured one of my gluster nodes and forgot to update the MTU >> size on the switch while I did configure the host with jumbo frames. >> >> The result was that the complete cluster had communication issues. >> >> All systems are part of a distributed striped volume with a replica size of >> 2 but still the cluster was completely unusable until I updated the switch >> port to accept jumbo frames rather than to discard them. > This is expected. When enabling the network components to communicate > with TCP jumbo frames in a Gluster Trusted Storage Pool, you'd need to > ensure that all the network components such as switches, nodes are > configured properly. I think with this setting you'd fail to ping the > other nodes in the pool. So that could be a step of verification before > you set the cluster up. >> >> The symptoms were: >> >> - Gluster clients had a very hard time reading the volume information and >> thus couldn’t do any filesystem ops on them. >> - The glusterfs servers could see each other (peer status) and a volume info >> command was ok, but a volume status command would not return or would return >> a “staging failed” error. >> >> I know MTU size mixing and don’t fragment bit’s can screw up a lot but why >> wasn’t that gluster peer just discarded from the cluster so that not all >> clients kept on communicating with it and causing all sorts of errors. > To answer this question, peer status & volume info are local operation > and doesn't incur N/W, so in this very same case you might see peer > status showing all the nodes are connected all though there is a > breakage, OTOH in status command originator node communicates with other > peers and hence it fails there. > > HTH, > Atin >> >> I use glusterFS 3.6.2 at the moment….. >> >> Kind regards >> Sander >> >> >> >> >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] MTU size issue?
LS, I recently reconfigured one of my gluster nodes and forgot to update the MTU size on the switch while I did configure the host with jumbo frames. The result was that the complete cluster had communication issues. All systems are part of a distributed striped volume with a replica size of 2 but still the cluster was completely unusable until I updated the switch port to accept jumbo frames rather than to discard them. The symptoms were: - Gluster clients had a very hard time reading the volume information and thus couldn’t do any filesystem ops on them. - The glusterfs servers could see each other (peer status) and a volume info command was ok, but a volume status command would not return or would return a “staging failed” error. I know MTU size mixing and don’t fragment bit’s can screw up a lot but why wasn’t that gluster peer just discarded from the cluster so that not all clients kept on communicating with it and causing all sorts of errors. I use glusterFS 3.6.2 at the moment….. Kind regards Sander signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] geo-replication settings: ignore-deletes
LS, I’ve created a replication session (cascaded btw) from our production cluster to a backup cluster with version 3.6.2 on CentOS 6.6 and when I want to deactivate the “ignore-deletes” option I get the following error: [root@b02-bkp-01 ]# gluster volume geo-replication bkp01gv0 b02-bkp-02::bkp02gv0 config '!ignore-deletes' Reserved option geo-replication command failed According to the docs this is the way to deactivate the option, but clearly it doesn’t work! How do I de-actvate this option as I clearly don’t want deletes to be ignored when syncing 50TB ….. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterFS 3.6.2 migrate data by remove brick command
Jiri, I’ve updated the op-version yesterday online without any problems, so I hope to be able to migrate my old bricks to the new tomorrow night without hassle using the remove brick command once all new bricks are added. My new bricks are smaller than the current ones but higher in number so I could’t use the replace brick in any case… thanks for the support…. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday On 16 Apr 2015, at 08:58, Jiri Hoogeveen j.hoogev...@bluebillywig.com wrote: Hi Sander, Sorry for not getting back to you. I guess, when you don’t use quota you do not need to run the scripts. I do not have any experience changing the op-version on a running glusterfs cluster. But looking at some threads, it should be possible to change it on a running glusterfs cluster. But I think, only when all clients are the same version has the server. And good luck this weekend. Grtz, Jiri On 14 Apr 2015, at 15:15, Sander Zijlstra sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl wrote: Jiri, thanks, I totally missed the op-version part as it’s not mentioned in the upgrade instructions as per the link you send. Actually I read that link and because I do not use quota I didn’t use that script either. Can I update the op-version when the volume is online and currently doing a rebalance or shall I stop the rebalance, set the new op-version and then start the rebalance again? many thanks for all the input…. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday On 14 Apr 2015, at 15:01, Jiri Hoogeveen j.hoogev...@bluebillywig.com mailto:j.hoogev...@bluebillywig.com wrote: Hi Sander, If I take a look at http://www.gluster.org/community/documentation/index.php/OperatingVersions http://www.gluster.org/community/documentation/index.php/OperatingVersions Then operating-version=2 is for glusterfs version 3.4, so I guess you still will be using the old style. I think this is useful http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6 http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6 And don’t forget to upgrade the clients also. Grtz, Jiri On 14 Apr 2015, at 14:14, Sander Zijlstra sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl wrote: Jiri, thanks for the information, I just commented on a question about op-version…. I upgraded all systems to 3.6.2 does this mean they all will use the correct op-version and will not revert to old style behaviour? Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday On 14 Apr 2015, at 14:11, Jiri Hoogeveen j.hoogev...@bluebillywig.com mailto:j.hoogev...@bluebillywig.com wrote: Hi Sander, Since version 3.6 the remove brick command migrates the data away from the brick being removed, right? It should :) https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/pdf/Administration_Guide/Red_Hat_Storage-3-Administration_Guide-en-US.pdf https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/pdf/Administration_Guide/Red_Hat_Storage-3-Administration_Guide-en-US.pdf page 100, is I think a good start. I think this is the most complete documentation. When I have replicated bricks (replica 2), I also need to do remove brick volume replica 2 brick1 brick2 …. , right? Yes, you need to remove both replica’s at the same time. Last but not least, is there any way to tell how long a “remove brick” will take when it’s moving the data? I have dual 10GB ethernet between the cluster members and the brick storage is a RAID-6 set which can read 400-600MB/sec without any problems. Depends on the size of the disk, the number of files and type of file. Network speed is less a issu, then the IO on the disks/brick. To migratie data from one disk to a other (is like self-healing) GlusterFS will do a scan of all files on the disk, which can cause a high IO on the disk. Because you had also some performance issues, when you added some bricks, I will expect the same issue with remove brick. So do this at night if possible. Grtz, Jiri On 14 Apr 2015, at 12:53, Sander Zijlstra sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl wrote
Re: [Gluster-users] how to check/fix underlaying partition error?
Hi Pavel, you can simply stop the glusterd service and run the fsck, it's similar to rebooting a server which is part of a replicated volume. If all is ok before you can simply take down one of the two and once it comes back online it will be heal each file which hasn't been copied allready. Do take care of any client which has the volume mounted using the server you take down; that will loose connection also. Met vriendelijke groet / kind regards, Sander Zijlstra Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl | www.surfsara.nl | - Original Message - From: Pavel Riha pavel.r...@trilogic.cz To: gluster-users@gluster.org Sent: Wednesday, 15 April, 2015 10:28:50 Subject: [Gluster-users] how to check/fix underlaying partition error? Hi guys, I have replicated glusterfs (v3.4.2) on two server and I found logs filled by IO error on one server only. But in /var/log/messages is no hw error, only XFS error, so I gues the filesystem could be corrupted My question is, how to stop or pause this brick and run fsck ? From the replicate feature I'm expecting no need to stop the gluster volume (there are some xen VM running) what is the right way to do it? with the later re-adding and fast rebuild/sync in mind.. thank for tips Pavel ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterFS 3.6.2 migrate data by remove brick command
Jiri, thanks for the information, I just commented on a question about op-version…. I upgraded all systems to 3.6.2 does this mean they all will use the correct op-version and will not revert to old style behaviour? Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday On 14 Apr 2015, at 14:11, Jiri Hoogeveen j.hoogev...@bluebillywig.com wrote: Hi Sander, Since version 3.6 the remove brick command migrates the data away from the brick being removed, right? It should :) https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/pdf/Administration_Guide/Red_Hat_Storage-3-Administration_Guide-en-US.pdf https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/pdf/Administration_Guide/Red_Hat_Storage-3-Administration_Guide-en-US.pdf page 100, is I think a good start. I think this is the most complete documentation. When I have replicated bricks (replica 2), I also need to do remove brick volume replica 2 brick1 brick2 …. , right? Yes, you need to remove both replica’s at the same time. Last but not least, is there any way to tell how long a “remove brick” will take when it’s moving the data? I have dual 10GB ethernet between the cluster members and the brick storage is a RAID-6 set which can read 400-600MB/sec without any problems. Depends on the size of the disk, the number of files and type of file. Network speed is less a issu, then the IO on the disks/brick. To migratie data from one disk to a other (is like self-healing) GlusterFS will do a scan of all files on the disk, which can cause a high IO on the disk. Because you had also some performance issues, when you added some bricks, I will expect the same issue with remove brick. So do this at night if possible. Grtz, Jiri On 14 Apr 2015, at 12:53, Sander Zijlstra sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl wrote: LS, I’m planning to decommission a few servers from my cluster, so to confirm: Since version 3.6 the remove brick command migrates the data away from the brick being removed, right? When I have replicated bricks (replica 2), I also need to do remove brick volume replica 2 brick1 brick2 …. , right? Last but not least, is there any way to tell how long a “remove brick” will take when it’s moving the data? I have dual 10GB ethernet between the cluster members and the brick storage is a RAID-6 set which can read 400-600MB/sec without any problems. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday ___ Gluster-users mailing list Gluster-users@gluster.org mailto:Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] glusterFS 3.6.2 migrate data by remove brick command
LS, I’m planning to decommission a few servers from my cluster, so to confirm: Since version 3.6 the remove brick command migrates the data away from the brick being removed, right? When I have replicated bricks (replica 2), I also need to do remove brick volume replica 2 brick1 brick2 …. , right? Last but not least, is there any way to tell how long a “remove brick” will take when it’s moving the data? I have dual 10GB ethernet between the cluster members and the brick storage is a RAID-6 set which can read 400-600MB/sec without any problems. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs 3.6.2: gluster volume add-brick: wronp op-version
LS, My cluster reads operating-version=2” on all nodes, and half of my nodes are fresh installs and the other halve I recently upgraded from 3.4 to 3.6.2. Does this mean that GlusterFS will not use the 3.6 versions of commands? Adding bricks from my new nodes to the volume already configured on the upgraded machines worked without a problem and currently the volume is still rebalancing. I’m about to do “remove brick” this weekend and I definitely need the data to be moved to the new bricks rather than the bricks be simply removed from the volume so will this go ok even though the op-version is 2?? Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl | www.surfsara.nl | Regular day off on friday On 14 Apr 2015, at 13:51, Kaushal M kshlms...@gmail.com wrote: You can get the current op-version by looking at /var/lib/glusterd/glusterd.info . GlusterFS-3.6 has no way to get it via a command. But you can set the via a command. Do `gluster volume set all op-version 30600` to set the op-version of the cluster to 30600, which is required for add-brick to work on 3.6. ~kaushal On Tue, Apr 14, 2015 at 5:16 PM, Gregor Burck gre...@aeppelbroe.de wrote: Hi, I don't understand what op-version stand for? OK I find the documentation: http://www.gluster.org/community/documentation/index.php/Features/Opversion I'm not sure If I tried to increase the op-version right: Im probe all exist nodes again? I still got the error message. In the existing cluster I've three nodes: edgf001, edgf002 and edgf003: gluster volume info samba1: Volume Name: samba1 Type: Replicate Volume ID: 4283053e-f50b-41c0-8b77-aa9985649b66 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: edgf001:/export/samba1 Brick2: edgf002:/export/samba1 Brick3: edgf003:/export/samba1 I installed the nodes as 3.5.X and made an upgrade sucessfull on 3.6.2 an all nodes. Where could I get the op-version from node? Bye Gregor ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How Can I Find Remaining Time for a Rebalance?
Ken, I have the same situation, last Sunday I started a rebalance after I added 3x18TB replicated bricks to a 2x37TB replicated brick volume which holds around 30 million files and I see that roughly 2 million files have been rebalanced with 2,5 TB data being moved.. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl | www.surfsara.nl | Regular day off on friday On 14 Apr 2015, at 00:02, Joe Julian j...@julianfamily.org wrote: Just as how you can't ask xfs or ext4 how many files it has, you can't ask Gluster the same thing. It doesn't know. It crawls the tree, looks at the hash of the filename, and determines if it should reside on a different brick. To guess how much remains, look at how many are complete out of 25 million. On 04/13/2015 02:19 PM, Ken Schweigert wrote: I've got a 4-node distributed-replicated cluster running 3.5.2. I just added 4TB (2TB to the distribute pair and 2TB to the replicate pair) and then started a rebalance as I believe the rebalance needed to be initiated for the extra space to be presented to the entire volume (correct me if I'm wrong). The rebalance has been running for about a week (it contains 12TB and about 25 million files) and I am unable to tell how much longer the rebalance will take. The output shows how long it's been running, but not how much time remains. Is there another command I can run to show how much remains? If not, is there any way to calculate how much time it has left, even if it's only a rough estimate? Thanks! -ken ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterFS 3.6.2 migrate data by remove brick command
Jiri, thanks, I totally missed the op-version part as it’s not mentioned in the upgrade instructions as per the link you send. Actually I read that link and because I do not use quota I didn’t use that script either. Can I update the op-version when the volume is online and currently doing a rebalance or shall I stop the rebalance, set the new op-version and then start the rebalance again? many thanks for all the input…. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday On 14 Apr 2015, at 15:01, Jiri Hoogeveen j.hoogev...@bluebillywig.com wrote: Hi Sander, If I take a look at http://www.gluster.org/community/documentation/index.php/OperatingVersions http://www.gluster.org/community/documentation/index.php/OperatingVersions Then operating-version=2 is for glusterfs version 3.4, so I guess you still will be using the old style. I think this is useful http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6 http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.6 And don’t forget to upgrade the clients also. Grtz, Jiri On 14 Apr 2015, at 14:14, Sander Zijlstra sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl wrote: Jiri, thanks for the information, I just commented on a question about op-version…. I upgraded all systems to 3.6.2 does this mean they all will use the correct op-version and will not revert to old style behaviour? Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday On 14 Apr 2015, at 14:11, Jiri Hoogeveen j.hoogev...@bluebillywig.com mailto:j.hoogev...@bluebillywig.com wrote: Hi Sander, Since version 3.6 the remove brick command migrates the data away from the brick being removed, right? It should :) https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/pdf/Administration_Guide/Red_Hat_Storage-3-Administration_Guide-en-US.pdf https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/pdf/Administration_Guide/Red_Hat_Storage-3-Administration_Guide-en-US.pdf page 100, is I think a good start. I think this is the most complete documentation. When I have replicated bricks (replica 2), I also need to do remove brick volume replica 2 brick1 brick2 …. , right? Yes, you need to remove both replica’s at the same time. Last but not least, is there any way to tell how long a “remove brick” will take when it’s moving the data? I have dual 10GB ethernet between the cluster members and the brick storage is a RAID-6 set which can read 400-600MB/sec without any problems. Depends on the size of the disk, the number of files and type of file. Network speed is less a issu, then the IO on the disks/brick. To migratie data from one disk to a other (is like self-healing) GlusterFS will do a scan of all files on the disk, which can cause a high IO on the disk. Because you had also some performance issues, when you added some bricks, I will expect the same issue with remove brick. So do this at night if possible. Grtz, Jiri On 14 Apr 2015, at 12:53, Sander Zijlstra sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl wrote: LS, I’m planning to decommission a few servers from my cluster, so to confirm: Since version 3.6 the remove brick command migrates the data away from the brick being removed, right? When I have replicated bricks (replica 2), I also need to do remove brick volume replica 2 brick1 brick2 …. , right? Last but not least, is there any way to tell how long a “remove brick” will take when it’s moving the data? I have dual 10GB ethernet between the cluster members and the brick storage is a RAID-6 set which can read 400-600MB/sec without any problems. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday ___ Gluster-users mailing list Gluster-users@gluster.org mailto:Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users http://www.gluster.org/mailman/listinfo/gluster-users signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.6.2 heal-failed info command gone?
Hi,
thanks.. this is very useful…..
Met vriendelijke groet / kind regards,
Sander Zijlstra
| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6
43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl
| www.surfsara.nl http://www.surfsara.nl/ |
Regular day off on friday
On 13 Apr 2015, at 16:23, Ravishankar N ravishan...@redhat.com wrote:
On 04/13/2015 07:20 PM, Sander Zijlstra wrote:
Hi,
thanks, I missed that one….
Does this mean that I can rely on looking at “volume heal info” output to
notice issues?
I regularly see those numbers return “0” so I can rely on that meaning that
no files failed healing?
That is correct. 'volume heal info' returns all entries that need healing,
including ones that are in split-brain. Those in split-brain are specifically
listed with Is in split-brain tag. 'volume heal info split-brain' lists
only the ones that are in split-brain.
Hope this helps,
Ravi
Off course as per the bug report I still need to check the “split-brain”
numbers…
thanks..
Met vriendelijke groet / kind regards,
Sander Zijlstra
| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31
(0)6 43 99 12 47 | sander.zijls...@surfsara.nl
mailto:sander.zijls...@surfsara.nl | www.surfsara.nl
http://www.surfsara.nl/ |
Regular day off on friday
On 13 Apr 2015, at 15:08, Ravishankar N ravishan...@redhat.com
mailto:ravishan...@redhat.com wrote:
On 04/13/2015 06:23 PM, Sander Zijlstra wrote:
LS,
I recently upgraded to 3.6.2 and I noticed that the info command for
heal-failed is not working:
# gluster volume heal gv0 info failed
Usage: volume heal VOLNAME [{full | statistics {heal-count {replica
hostname:brickname}} |info {healed | heal-failed | split-brain}}]
[root@v39-app-01 ~]# gluster volume heal gv0 info heal-failed
Command not supported. Please use gluster volume heal gv0 info and logs
to find the heal information.
The documentation on GitHub says “gluster volume vol info failed” as
command but that’s clearly no correct, and when asking for “heal-failed” I
get a funny response…..
The other two options, “info” and “info split-brain” perform as expected,
only the “failed” not…
any ideas why?
'info healed ' and ' info heal-failed' are not supported any more since
http://review.gluster.org/#/c/7766/ http://review.gluster.org/#/c/7766/
Please see #2 in https://bugzilla.redhat.com/show_bug.cgi?id=1098027#c0
https://bugzilla.redhat.com/show_bug.cgi?id=1098027#c0
-Ravi
Met vriendelijke groet / kind regards,
Sander Zijlstra
| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31
(0)6 43 99 12 47 | sander.zijls...@surfsara.nl
mailto:sander.zijls...@surfsara.nl | www.surfsara.nl
http://www.surfsara.nl/ |
Regular day off on friday
___
Gluster-users mailing list
Gluster-users@gluster.org mailto:Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
http://www.gluster.org/mailman/listinfo/gluster-users
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] 3.6.2 heal-failed info command gone?
LS,
I recently upgraded to 3.6.2 and I noticed that the info command for
heal-failed is not working:
# gluster volume heal gv0 info failed
Usage: volume heal VOLNAME [{full | statistics {heal-count {replica
hostname:brickname}} |info {healed | heal-failed | split-brain}}]
[root@v39-app-01 ~]# gluster volume heal gv0 info heal-failed
Command not supported. Please use gluster volume heal gv0 info and logs to
find the heal information.
The documentation on GitHub says “gluster volume vol info failed” as command
but that’s clearly no correct, and when asking for “heal-failed” I get a funny
response…..
The other two options, “info” and “info split-brain” perform as expected, only
the “failed” not…
any ideas why?
Met vriendelijke groet / kind regards,
Sander Zijlstra
| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6
43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl
| www.surfsara.nl http://www.surfsara.nl/ |
Regular day off on friday
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.6.2 heal-failed info command gone?
Hi,
thanks, I missed that one….
Does this mean that I can rely on looking at “volume heal info” output to
notice issues?
I regularly see those numbers return “0” so I can rely on that meaning that no
files failed healing?
Off course as per the bug report I still need to check the “split-brain”
numbers…
thanks..
Met vriendelijke groet / kind regards,
Sander Zijlstra
| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6
43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl
| www.surfsara.nl http://www.surfsara.nl/ |
Regular day off on friday
On 13 Apr 2015, at 15:08, Ravishankar N ravishan...@redhat.com wrote:
On 04/13/2015 06:23 PM, Sander Zijlstra wrote:
LS,
I recently upgraded to 3.6.2 and I noticed that the info command for
heal-failed is not working:
# gluster volume heal gv0 info failed
Usage: volume heal VOLNAME [{full | statistics {heal-count {replica
hostname:brickname}} |info {healed | heal-failed | split-brain}}]
[root@v39-app-01 ~]# gluster volume heal gv0 info heal-failed
Command not supported. Please use gluster volume heal gv0 info and logs to
find the heal information.
The documentation on GitHub says “gluster volume vol info failed” as
command but that’s clearly no correct, and when asking for “heal-failed” I
get a funny response…..
The other two options, “info” and “info split-brain” perform as expected,
only the “failed” not…
any ideas why?
'info healed ' and ' info heal-failed' are not supported any more since
http://review.gluster.org/#/c/7766/ http://review.gluster.org/#/c/7766/
Please see #2 in https://bugzilla.redhat.com/show_bug.cgi?id=1098027#c0
https://bugzilla.redhat.com/show_bug.cgi?id=1098027#c0
-Ravi
Met vriendelijke groet / kind regards,
Sander Zijlstra
| Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31
(0)6 43 99 12 47 | sander.zijls...@surfsara.nl
mailto:sander.zijls...@surfsara.nl | www.surfsara.nl
http://www.surfsara.nl/ |
Regular day off on friday
___
Gluster-users mailing list
Gluster-users@gluster.org mailto:Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
http://www.gluster.org/mailman/listinfo/gluster-users
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] GlusterFS 3.6.2, volume from 4 to 8 bricks CPU went sky high
LS, We have a GlusterFS cluster which consists of 4 nodes with one brick each and a distributed-replicated volume of 72 TB. Today I extended the cluster to 8 machines and added new bricks to the volume, so it now contains 8 bricks. I didn’t start the rebalance yet to limit the impact during the day but to my surprise all glusterfsd process went sky high and performance was really really bad. So effectively I cause downtime to our storage service while I didn’t anticipated this, hence I didn’t do any rebalance yet. Can somebody explain to me why adding bricks to a volume causes this high CPU usage? I can imagine the meta data needed to be synced but if this is so heavy, why can’t I tune this? This is my current volume setup: Volume Name: gv0 Type: Distributed-Replicate Volume ID: 0322f20f-e507-492b-91db-cb4c953a24eb Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: s-s35-06:/glusterfs/bricks/brick1/brick Brick2: s-s35-07:/glusterfs/bricks/brick1/brick Brick3: s-s35-08:/glusterfs/bricks/brick1/brick Brick4: s-s35-09:/glusterfs/bricks/brick1/brick Brick5: v39-app-01:/glusterfs/bricks/brick1/gv0 Brick6: v39-app-02:/glusterfs/bricks/brick1/gv0 Brick7: v39-app-03:/glusterfs/bricks/brick1/gv0 Brick8: v39-app-04:/glusterfs/bricks/brick1/gv0 Options Reconfigured: performance.cache-size: 256MB nfs.disable: on geo-replication.indexing: off geo-replication.ignore-pid-check: on changelog.changelog: on performance.io-thread-count: 32 performance.write-behind-window-size: 5MB Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] meta_volume -- Invalid option?
LS, I tried to add a meta volume to my geo-replication session but the command doesn’t seem to be recognised : [root@s35-06 glusterfs]# gluster volume geo-replication gv0 v39-app-01::gv0 config meta_volume georep_metavol Invalid option geo-replication command failed [root@s35-06 glusterfs]# gluster volume info georep_metavol Volume Name: georep_metavol Type: Replicate Volume ID: f048fe24-d0e7-44aa-9d07-10a775abfe0f Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: s-s35-06:/glusterfs/bricks/brick1/georep_metavol Brick2: s-s35-07:/glusterfs/bricks/brick1/georep_metavol Brick3: s-s35-08:/glusterfs/bricks/brick1/georep_metavol I used this documentation: https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.6.2 geo-replication session created but no files are copied
Aravinda, I wasn’t really referring to the SSH command itself but the error thrown by gsyncd.py: error: incorrect number of arguments Usage: gsyncd.py [options...] master slave The status is still like this: # gluster volume geo-replication gv0 v39-app-01::gv0 status MASTER NODE MASTER VOLMASTER BRICK SLAVESTATUS CHECKPOINT STATUS CRAWL STATUS - s35-06gv0 /glusterfs/bricks/brick1/brickv39-app-05::gv0 Active N/A Changelog Crawl s35-07 gv0 /glusterfs/bricks/brick1/brickv39-app-02::gv0 PassiveN/A N/A s35-08gv0 /glusterfs/bricks/brick1/brickv39-app-01::gv0 Active N/A Changelog Crawl s35-09 gv0 /glusterfs/bricks/brick1/brickv39-app-04::gv0 PassiveN/A N/A Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl | www.surfsara.nl | Regular day off on friday On 08 Apr 2015, at 05:39, Aravinda avish...@redhat.com wrote: I don't see any issue with ssh command here. Geo-rep will use this command as prefix and adds additional parameters while running. Please let us know the Current status and any errors in log files(/var/log/glusterfs/geo-replication/MASTERVOL_SLAVEHOST_SLAVEVOL/) -- regards Aravinda On 04/08/2015 12:18 AM, Sander Zijlstra wrote: LS, Last week I configured geo-replication between two glusterFS clusters, both version 3.6.2 and all looks ok: [root@s35-06 gv0]# gluster volume geo-replication gv0 status MASTER NODE MASTER VOLMASTER BRICK SLAVE STATUS CHECKPOINT STATUSCRAWL STATUS -- s35-06 gv0 /glusterfs/bricks/brick1/brick v39-app-05::gv0 Active N/A Changelog Crawl s35-07 gv0 /glusterfs/bricks/brick1/brick v39-app-02::gv0 PassiveN/A N/A s35-09 gv0 /glusterfs/bricks/brick1/brick v39-app-04::gv0 PassiveN/A N/A s35-08 gv0 /glusterfs/bricks/brick1/brick v39-app-01::gv0 Active N/A Changelog Crawl I started the replication at the end of the day, hoping that all 40TB would be copied by the next day or so, but I discovered that not a single bit has been copied. When looking at the volume config settings I found the “ssh command” used so I tried that and discovered the following issue between my master and slave cluster when executing the configured “ssh command [root@s35-06 gv0]# ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem v39-app-01 [2015-04-07 14:43:43.613130] I [cli.c:593:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost [2015-04-07 14:43:43.613178] D [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2015-04-07 14:43:43.613187] D [rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2015-04-07 14:43:43.613202] D [rpc-transport.c:188:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to socket [2015-04-07 14:43:43.613211] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so [2015-04-07 14:43:43.615501] T [options.c:87:xlator_option_validate_int] 0-glusterfs: no range check required for 'option remote-port 24007' [2015-04-07 14:43:43.615528] D [socket.c:3799:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2015-04-07 14:43:43.615537] D [socket.c:3802:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2015-04-07 14:43:43.615543] D [socket.c:3819:socket_init] 0-glusterfs: using system polling thread ——%——— [2015-04-07 14:43:43.733052] I [cli-rpc-ops.c:5386:gf_cli_getwd_cbk] 0-cli: Received resp to getwd [2015-04-07 14:43:43.733085] D [cli-cmd.c:384:cli_cmd_submit] 0-cli: Returning 0 [2015-04-07 14:43:43.733097] D [cli-rpc-ops.c:5415:gf_cli_getwd] 0-cli: Returning 0 [2015-04-07 14:43:43.733104] I [input.c:36:cli_batch] 0-: Exiting with: 0 error: incorrect number of arguments Usage: gsyncd.py [options...] master slave Connection to v39-app-01 closed. Can somebody point me to how to fix this “gsycnd” issue? I didn’t find any updated packages from CentOS for my release (6.6), so I expect it should be a working setup. any help would appreciated
Re: [Gluster-users] meta_volume -- Invalid option?
Hi, Aha, then I’m mistaken as I’m using 3.6.2 thanks…. Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl | www.surfsara.nl | Regular day off on friday On 08 Apr 2015, at 13:38, Aravinda avish...@redhat.com wrote: Hi, Please let us know the Gluster version you are using. This is a recent enhancement which will be available in the upcoming Gluster 3.7 release. -- regards Aravinda http://aravindavk.in On 04/08/2015 04:19 PM, Sander Zijlstra wrote: LS, I tried to add a meta volume to my geo-replication session but the command doesn’t seem to be recognised : [root@s35-06 glusterfs]# gluster volume geo-replication gv0 v39-app-01::gv0 config meta_volume georep_metavol Invalid option geo-replication command failed [root@s35-06 glusterfs]# gluster volume info georep_metavol Volume Name: georep_metavol Type: Replicate Volume ID: f048fe24-d0e7-44aa-9d07-10a775abfe0f Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: s-s35-06:/glusterfs/bricks/brick1/georep_metavol Brick2: s-s35-07:/glusterfs/bricks/brick1/georep_metavol Brick3: s-s35-08:/glusterfs/bricks/brick1/georep_metavol I used this documentation: https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_distributed_geo_rep.md Met vriendelijke groet / kind regards, *Sander Zijlstra* | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl | /Regular day off on friday/ ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] 3.6.2 geo-replication session created but no files are copied
LS, Last week I configured geo-replication between two glusterFS clusters, both version 3.6.2 and all looks ok: [root@s35-06 gv0]# gluster volume geo-replication gv0 status MASTER NODE MASTER VOLMASTER BRICK SLAVE STATUS CHECKPOINT STATUSCRAWL STATUS -- s35-06gv0 /glusterfs/bricks/brick1/brickv39-app-05::gv0 Active N/A Changelog Crawl s35-07gv0 /glusterfs/bricks/brick1/brickv39-app-02::gv0 PassiveN/A N/A s35-09gv0 /glusterfs/bricks/brick1/brickv39-app-04::gv0 PassiveN/A N/A s35-08gv0 /glusterfs/bricks/brick1/brickv39-app-01::gv0 Active N/A Changelog Crawl I started the replication at the end of the day, hoping that all 40TB would be copied by the next day or so, but I discovered that not a single bit has been copied. When looking at the volume config settings I found the “ssh command” used so I tried that and discovered the following issue between my master and slave cluster when executing the configured “ssh command [root@s35-06 gv0]# ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem v39-app-01 [2015-04-07 14:43:43.613130] I [cli.c:593:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost [2015-04-07 14:43:43.613178] D [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2015-04-07 14:43:43.613187] D [rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2015-04-07 14:43:43.613202] D [rpc-transport.c:188:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to socket [2015-04-07 14:43:43.613211] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so [2015-04-07 14:43:43.615501] T [options.c:87:xlator_option_validate_int] 0-glusterfs: no range check required for 'option remote-port 24007' [2015-04-07 14:43:43.615528] D [socket.c:3799:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2015-04-07 14:43:43.615537] D [socket.c:3802:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2015-04-07 14:43:43.615543] D [socket.c:3819:socket_init] 0-glusterfs: using system polling thread ——%——— [2015-04-07 14:43:43.733052] I [cli-rpc-ops.c:5386:gf_cli_getwd_cbk] 0-cli: Received resp to getwd [2015-04-07 14:43:43.733085] D [cli-cmd.c:384:cli_cmd_submit] 0-cli: Returning 0 [2015-04-07 14:43:43.733097] D [cli-rpc-ops.c:5415:gf_cli_getwd] 0-cli: Returning 0 [2015-04-07 14:43:43.733104] I [input.c:36:cli_batch] 0-: Exiting with: 0 error: incorrect number of arguments Usage: gsyncd.py [options...] master slave Connection to v39-app-01 closed. Can somebody point me to how to fix this “gsycnd” issue? I didn’t find any updated packages from CentOS for my release (6.6), so I expect it should be a working setup. any help would appreciated Met vriendelijke groet / kind regards, Sander Zijlstra | Linux Engineer | SURFsara | Science Park 140 | 1098XG Amsterdam | T +31 (0)6 43 99 12 47 | sander.zijls...@surfsara.nl mailto:sander.zijls...@surfsara.nl | www.surfsara.nl http://www.surfsara.nl/ | Regular day off on friday signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
