Re: [Gluster-users] moving drives containing bricks from one server to another
It would help if you say what kind of volumes you have, as they all work a little differently. You should NOT destroy the old volumes. It should be quite possible to move the bricks to a new server and get them running as part of THE SAME gluster system that you have now. Until you tell us how many bricks, how many servers have what bricks, exactly which servers you are replacing (you mention only one server, but that is not a normal gluster configuration), we can't give you a whole lot of help. The usual routine involves working with one brick server at a time, while keeping the rest of the array untouched. With some replicated arrays and some careful work, you don't have to take the array offline at all. However, replicated arrays get handled VERY differently from distributed, so we can't help much until you tell us more. Ted Miller Elkhart, IN On 07/18/2017 06:18 AM, Andy Tai wrote: hi, I did not see a reply to my problem. Let me ask it in a different way... If I have bricks from a previous glusterfs volume and that volume is now gone because of the old machine was replaced, now I tried to create a new volume and add the old bricks to the new volume with the "force" opinion to "volume create". The old data files are still in the bricks but when I mount the new volume the new volume shows it is empty. Is it possible for glusterfs to recognize the old data files on the bricks in some way to essentially re-create the same view as the old volume, via some kind of healing or meta data re-scan or re-creation? The bricks are in good shape with no data corruption and are in sync (same data were replicated). Thanks On Thu, May 4, 2017 at 10:03 PM, Andy Tai <a...@atai.org> wrote: Hi, I have a gluster volume with bricks spread over several physical drives. I now want to upgrade my server to a new system and plan to move the drives from the old server to the new server, with a different host name and IP address. Can I shut down the gluster volume on the old server, move and install the physical drives containing the bricks to the new server, and then create a new gluster volume on the new server, and add the bricks to the new volume in the same way reflecting the previous organization of the volume on the old server, and expect everything to work (all files preserved and accessible via glusterfs, with no data loss? The gluster volume on the old server would be retired and I want to let the new server taking over the role of serving the gluster volume. Thanks -- Andy Tai, a...@atai.org, Skype: licheng.tai, Line: andy_tai, WeChat: andytai1010 Year 2017 民國106年 自動的精神力是信仰與覺悟 自動的行為力是勞動與技能 -- Andy Tai, a...@atai.org,
Re: [Gluster-users] Moving a physical disk from one machine to another
On 02/06/2017 11:11 AM, Scott Hazelhurst wrote: Apologies if this is obvious but I’ve searched the documentation and mailing list and can’t see an answer. One of my servers which hosts a gluster brick has died beyond possible saving. The physical disk of the brick itself is good and I’d like not to lose anything. Can I just add it as a brick on another server? Unfortunately, I am not able to host the disk on a machine with the same IP address/host name. I suppose I could just add a new empty disk and then copy the data from the old disk but I’d prefer to do things as easily as possible. Scott This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. http://lists.gluster.org/mailman/listinfo/gluster-users Did this a few months ago, when I had a server die. Assuming it is a replicated volume, first thing is to remove old brick from your volume. It has been a while since I had to do this, but I think you just do a "gluster remove-brick ". It will probably require a 'force' since the server is gone. You should be able to install the disk in another server (with gluster installed). Then just add that brick to the volume, like you did when setting up the volume. You will again have to force, because the volume will already have a volume signature. After that do a full heal, and wait. It will save a good bit of time that the files are already on the disk. It should add any files that have been changed, and then go on its way. You may want to be sure that any files deleted since the server died don't get put back into the volume. Have had that happen, but don't remember if this was the combination that triggered it, or something else. Ted Miller Indiana, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Mount gluster volume inside other mounted volume...
On 08/25/2016 08:11 AM, Gilberto Nunes wrote: Hello list I have two volumes, DATA and WORK. DATA has size 500 GB WORK has size 1,2 TB I can mount DATA with this command: mount -t glusterfs -o acl localhost:DATA /home Everything is ok with that. But, when I mont WORK volume inside /home/work, like this mount -t glusterfs -o acl localhost:WORK /home/work I realize that /home and /home/work has the same size: df -h localhost:WORK 539G 45M 539G 1% /home/work localhost:DATA 539G 45M 539G 1% /home Are the bricks for WORK and DATA both subdirectories in the same partition on at one of your computers? If the bricks are just subdirectories, rather than dedicated partitions, df will look like that. I have some "odd" volumes that are subdirectories (for now) and they look like that. Anything I add to either volume will increase the USED and decrease the AVAILABLE for both volumes. What df shows is the figures for the partition that these are part of, not for the particular subdirectory that you turned into a brick. If that isn't your setup, someone else will have to help you. Ted Miller Elkhart, IN, USA Is there a way to workaround this?? Perhaps this is no a issue related to glusterfs, but I need just so advice where found a possible solution Thanks anyway Best regards Gilberto Ferreira ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] 'nofail' fails miserably
Now that most distros are switching to systemd, there seems to be a problem in the mount command's handling of the 'nofail' option. Setup: using glusterfs 3.7.13 on Centos 7 (up to date) from the Centos Storage SIG repo. The man page for systemd.mount says: nofail With nofail this mount will be only wanted, not required, by local-fs.target or remote-fs.target. This means that the boot will continue even if this mount point is not mounted successfully. It works fine during bootup -- things end up mounted, but they don't timeout and throw the server into maintenance mode if they are a little slow. However, if I do a 'mount -a' from the command line, and any gluster volume needs to be mounted, mount (I assume mount.glusterfs) throws the response: Invalid option nofail Somebody needs to take responsibility for either filtering out the 'nofail' option so that mount.glusterfs doesn't see it, or else glusterfs needs to be smart enough to recognize that, even though it is in the options space, it is OK to ignore it. If it is OK for systemd, and the only place you can put it is in the mount options, then mount.glusterfs needs to be OK with that. I have not tried the other options that systemd.mount uses, so some of them may also cause problems. man systemd.mount lists these options as being used: x-systemd.requires= x-systemd.requires-mounts-for= x-systemd.automount x-systemd.device-timeout= noauto auto nofail x-initrd.mount 'noauto' has been around forever, so it is probably handled OK, (I think I used it a while back) but not all the new options are handled right, or else the documentation is bad. I hope somebody can stick a couple of lines of code in so we can mount 'nofail' volumes after boot. Thank you, Ted Miller Elkhart, Indiana, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster replica over WAN...
On 8/2/2016 8:14 AM, Gilberto Nunes wrote: Hello list... This is my first post on this list. I have here two IBM Server, with 9 TB of hardisk on which one. Between this servers, I have a WAN connecting two offices,let say OFFICE1 and OFFICE2. This WAN connection is over fibre channel. When I setting up gluster with replica with two bricks, and mount the gluster volume in other folder, like this: mount -t glusterfs localhost:/VOLUME /STORAGE and when I go to that folder, and try to access the content, I get a lot of timeout... Even a single ls give a lot of time to return the list. This folder, /STORAGE is access by many users, through samba share. So, when OFFICE1 access the gluster server access the files over \\server\share, has a long delay to show the files Sometimes, get time out. My question is: is there some way to improve the gluster to work faster in this scenario?? Thanks a lot. Best regards -- Gilberto Ferreira +55 (47) 9676-7530 Skype: gilberto.nunes36 ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users I probably can't help, but I can tell you the kind of things that will help people to understand your problem better. What is the bandwidth between sites? What is the ping time between sites? If you run "ping -c1 " from OFFICE1, how many pings fail? What else can you tell us about the WAN connection that would help us understand your situation? How many files are there in the \\server\share directory? Are people at both sites actively writing files to the shared storage? If not, you may need to look at gluster geo-replication. It is one way, so all writing would have to be done to OFFICE1, with OFFICE2 having read-only access. (Some things that works wonderfully for, other things it doesn't work at all.) I can also tell you that no one who has tried it will endorse running a production replica 2 cluster over WAN. You are asking for either downtime or split-brain or both. Replica 3 is the minimum for production use of a replicate cluster, even over LAN. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.13 two node ssd solid rock
On 8/3/2016 11:13 AM, Leno Vo wrote: One of my gluster 3713 is on two nodes only with samsung ssd 1tb pro raid 5 x3,it already crashed two time because of brown out and block out, it got production vms on it, about 1.3TB. Never got split-brain, and healed quickly. Can we say 3.7.13 two nodes with ssd is solid rock or just lucky? My other gluster is on 3 nodes 3713, but one node never got up (old server proliant wants to retire), ssh raid 5 with combination sshd lol laptop seagate, it never healed about 586 occurences but there's no split-brain too. and vms are intact too, working fine and fast. ahh never turned on caching on the array, the esx might not come up right away, u need to go to setup first to make it work and restart and then you can go to array setup (hp array F8) and turned off caching. then esx finally boot up. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users I would say you are very lucky. I would not use anything less that replica 3 in production. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Problem with Glusterfs - taking over CPU time on clients
On 7/1/2016 5:22 AM, Milos Kurtes wrote: Hi, I have 7 clients on AWS in ELB and every of them is connected to glusterfs servers group of two. On clients, version of glusterfs is glusterfs 3.7.1 built on Nov 22 2015 17:39:20 and on server side both of servers have same glusterfs 3.6.2 built on Jan 22 2015 12:58:10 The major and urgent problem is the situation after disconnecting and connecting one of the servers, on one of the clients (and that is changing which one after few minutes) are overtaken with glusterfs process (all apache processes are on sleep in the same moment) which is responsible for rw session files. This is a very frequent job. Without disconnection, everything working except filling log files with a big amount of messages (probably incompatibility versions of glusterfs on servers and clients side why I tried yesterday to get the update for server side). Do anybody have a solution for this overtaking situation? Milos Kurtes, System Administrator, Alison Co. I'm no expert, but it sounds like the heal process hogging resources healing (or checking for differences) the replicated volume back into the cluster. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] About Gluster cluster availability when one of out of two nodes is down
Is it not the default behavior that if a volume looses quorum, the files are still available in read-only mode? If so, it makes sense to me that this behavior would continue after a reboot. Otherwise the user is going to be very confused. Ted Miller Elkhart, IN, USA On 6/30/2016 2:10 AM, Atin Mukherjee wrote: Currently on a two node set up, if node B goes down and node A is rebooted brick process(es) on node A doesn't come up to avoid split brains. However we have had concerns/bugs from different gluster users on the availability with this configuration. So we can solve this issue by starting the brick process(es) if quorum is not enabled. If quorum is enabled we'd not. Although quorum option really doesn't make sense in a two node cluster, but we can leverage this option to get rid of this specific situation. I'd like to know your feedback on this and then I push a patch right away. ~Atin ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] selinux status on RHEL/Centos 7
What is the status of selinux tagging on Centos 7? I have read enough to know that this is a chain-like process requiring changes in the client, the server, FUSE, and the kernel to make it all work. What is the current status of this process on Centos 7? My use-case: I need to allow Apache to access files that are stored on gluster and mounted using FUSE. What are my options (besides shutting down selinux for the Apache process)? Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] reading from local replica?
On 6/8/2015 5:55 PM, Brian Ericson wrote: Am I misunderstanding cluster.read-subvolume/cluster.read-subvolume-index? I have two regions, "A" and "B" with servers "a" and "b" in, respectfully, each region. I have clients in both regions. Intra-region communication is fast, but the pipe between the regions is terrible. I'd like to minimize inter-region communication to as close to glusterfs write operations only and have reads go to the server in the region the client is running in. I have created a replica volume as: gluster volume create gv0 replica 2 a:/data/brick1/gv0 b:/data/brick1/gv0 force As a baseline, if I use scp to copy from the brick directly, I get -- for a 100M file -- times of about 6s if the client scps from the server in the same region and anywhere from 3 to 5 minutes if I the client scps the server in the other region. I was under the impression (from something I read but can't now find) that glusterfs automatically picks the fastest replica, but that has not been my experience; glusterfs seems to generally prefer the server in the other region over the "local" one, with times usually in excess of 4 minutes. I've also tried having clients mount the volume using the "xlator" options cluster.read-subvolume and cluster.read-subvolume-index, but neither seem to have any impact. Here are sample mount commands to show what I'm attempting: mount -t glusterfs -o xlator-option=cluster.read-subvolume=gv0-client-<0 or 1> a:/gv0 /mnt/glusterfs mount -t glusterfs -o xlator-option=cluster.read-subvolume-index=<0 or 1> a:/gv0 /mnt/glusterfs Am I misunderstanding how glusterfs works, particularly when trying to "read locally"? Is it possible to configure glusterfs to use a local replica (or the "fastest replica") for reads? I am not a developer, nor intimately familiar with the insides of glusterfs, but here is how I understand that glusterfs-fuse file reads work. First, all replica bricks are read, to make sure they are consistent. (If not, gluster tries to make them consistent before proceeding). After consistency is established, then the actual read occurs from the brick with the shortest response time. I don't know when or how the response time is measured, but it seems to work for most people most of the time. (If the client is on one of the brick hosts, it will almost always read from the local brick.) If the file reads involve a lot of small files, the consistency check may be what is killing your response times, rather than the read of the file itself. Over a fast LAN, the consistency checks can take many times the actual read time of the file. Hopefully others will chime in with more information, but if you can supply more information about what you are reading, that will help too. Are you reading entire files, or just reading in a lot of "snippets" or what? Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] The strange behavior whose common denominator is gluster
On 6/9/2015 8:30 AM, Pablo Silva wrote: Hi Ted! Thanks for your reply, for the implementation of Service N2 (As2 Mendelson B45 + gluster 3.3.1-15.el6.x86_64 (1 brick) we have implemented one brick As you can see, Mendelson has a Brick, and use it for to process teh Purchase Ordes feed in SOA21 by MULE process [root@mendelson ~]# gluster volume info Volume Name: m2mas2 Type: Distribute Volume ID: 328d1e48-bb0a-4b41-92ac-699f78d2dca7 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: mendelson.X.b2b:/opt/mendelson/messages Options Reconfigured: auth.allow: 10.200.20.*,10.200.22.* storage.owner-uid: 500 storage.owner-gid: 500 [root@mendelson ~]# cat /etc/fstab # # /etc/fstab # Created by anaconda on Wed Oct 23 11:42:22 2013 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/VolGroup-lv_root / ext4defaults,acl1 1 UUID=df46586a-2f06-4ada-b2e6-523f7ec3967b /boot ext4 defaults1 2 /dev/mapper/VolGroup-lv_swap swap swapdefaults0 0 tmpfs /dev/shm tmpfs defaults0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults0 0 proc/proc procdefaults0 0 Here is your problem. You are not mounting m2mas2. /opt/mendelson/messages is NOT the same as m2mas2. You can NEVER write (or read) to /opt/mendelson/messages. You must pretend that is _does not exist_. You must mount the Gluster volume, and use only the path to the gluster volume, never to the brick. When you write to the brick you are "going behind gluster's back" and gluster does not know that anything has changed, so it does not make those files available to the gluster clients. It may eventually notice them, and make them available, but it is not anything dependable or predictable. First, make sure you have the glusterfs-fuse package installed. You need to add a line to /etc/fstab that is like: mendelson:/m2mas2/opt/m2m.as2 glusterfs defaults 0 0 Then change everything that you are now reading and writing to /opt/mendelson/messages so that it writes to /opt/m2m.as2 (or whatever mount point you have chosen). Many of us follow this pattern. All bricks are located at /bricks/xxx/xxx. All gluster files systems are mounted to /gluster/xxx. That makes the separation between the bricks and the gluster volumes very clear, and very hard to mess up when writing file paths in programs. No trying to remember "Which is the brick, and which is the gluster volume?" On your Apache system, you again need to mount the glusterfs volume in a separate place, and make sure that the glusterfs location is the only one that Apache knows about or uses in any way whatsoever. Ted Miller In this machine run MULE process for to feed with Purchase Orders to Mendelson. [root@soa21ifh ~]# mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw,acl) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/xvda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) glusterfs#mendelson:/m2mas2 on /opt/m2m.as2 type fuse (rw,allow_other,max_read=131072) maybe you.'re right and we are wrong with the way of architecture Thanks in Advance -Pablo On Mon, Jun 8, 2015 at 6:06 PM, Ted Miller <mailto:tmil...@sonsetsolutions.org>> wrote: Are you sure you have mounted the gluster volume, and are writing to the gluster volume, and NOT to the brick? What you describe can happen when you write to the brick instead of the gluster volume. You can see here: http://www.gluster.org/community/documentation/index.php/QuickStart in steps 6 and 7. If you do not understand the difference, include the output of the 'mount' command from one of your servers. Ted Miller Elkhart, IN, USA On 6/5/2015 8:46 AM, Pablo Silva wrote: Dear Colleagues: We are using gluster in 3.3.1-15.el6.x86_64 and GlusterFS-3.6.2-1.el5 versions, we have two types of service: 1) Apache httpd-2.2.3-91.el5.centos + GlusterFS-3.6.2-1.el5 (two bricks) 2) AS2 Mendelson B45 + gluster 3.3.1-15.el6.x86_64 (two bricks) It is different services, a common problem, which I will explain Service N1 (Apache httpd-2.2.3-91.el5.centos + GlusterFS-3.6.2-1.el5 (two bricks)) --- We have a high-availability architecture, in which there are two Apache servers see a directory that is hosted on a gluster long ago we had a problem where an Apache s
Re: [Gluster-users] nfs-ganesha/samba vfs and replica redundancy
On 6/3/2015 3:15 AM, Benjamin Kingston wrote: Can someone give me a hint on the best way to maintain data availability to a share on a third system using nfs-ganesha and samba? I currently have a round-robbin dns entry that nfs ganesha/samba uses, however even with a short ttl, there's brief downtime when a replica node fails. I can't see in the samba VFS or ganesha fsal syntax where a secondary address can be provided. I've tried comma seperated, space seperated, with/without quotes for multiple IP's and only seen issues. Any reason you aren't using a "floating IP address"? This isn't the newest talk, but the concepts have not changed: http://events.linuxfoundation.org/sites/events/files/lcjpcojp13_nakai.pdf Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] The strange behavior whose common denominator is gluster
Are you sure you have mounted the gluster volume, and are writing to the gluster volume, and NOT to the brick? What you describe can happen when you write to the brick instead of the gluster volume. You can see here: http://www.gluster.org/community/documentation/index.php/QuickStart in steps 6 and 7. If you do not understand the difference, include the output of the 'mount' command from one of your servers. Ted Miller Elkhart, IN, USA On 6/5/2015 8:46 AM, Pablo Silva wrote: Dear Colleagues: We are using gluster in 3.3.1-15.el6.x86_64 and GlusterFS-3.6.2-1.el5 versions, we have two types of service: 1) Apache httpd-2.2.3-91.el5.centos + GlusterFS-3.6.2-1.el5 (two bricks) 2) AS2 Mendelson B45 + gluster 3.3.1-15.el6.x86_64 (two bricks) It is different services, a common problem, which I will explain Service N1 (Apache httpd-2.2.3-91.el5.centos + GlusterFS-3.6.2-1.el5 (two bricks)) --- We have a high-availability architecture, in which there are two Apache servers see a directory that is hosted on a gluster long ago we had a problem where an Apache server could list the files and submit them for download, while the other Apache server that is watching the same directory with the same files gluster indicated that there were no files for download. Feeding gluster files to that directory, MULE performed asynchronously.In summary, an Apache server could access files and another did not give aware of their existence, as the directory and the same files. Service N2 (As2 Mendelson B45 + gluster 3.3.1-15.el6.x86_64 (two bricks) ) -- We have only one Mendelson AS2 Server B45 running with gluster (two bricks), The operations of mendelson is quite simple, is to observe the presence of files in a directory every 5 seconds and sent to the partner, the directory is hosted in gluster, the issue that every certain amount of time not Mendelson AS2 takes cognizance the existence of files in the directory, even if you enter the directory notes of its existence In both cases, different services being the only common denominator is gluster, someone else is experiencing this problem? Have we not set the service gluster well and we are repeating the same mistake ?, or is it a bug? Thanks in advance Pablo ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -- *Ted Miller*, Design Engineer *SonSet Solutions* (formerly HCJB Global Technology Center) my desk +1 574.970.4272 receptionist +1 574.972.4252 http://sonsetsolutions.org /Technology for abundant life!/ ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster 3.7.0 released
responses below Ted Miller On 5/26/2015 12:01 AM, Atin Mukherjee wrote: On 05/26/2015 03:12 AM, Ted Miller wrote: From: Niels de Vos Sent: Monday, May 25, 2015 4:44 PM On Mon, May 25, 2015 at 06:49:26PM +, Ted Miller wrote: From: Humble Devassy Chirammal Sent: Monday, May 18, 2015 9:37 AM Hi All, GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are available at download.gluster.org<http://download.gluster.org> [1]. [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/ --Humble On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur mailto:vbel...@redhat.com>> wrote: Hi All, I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 contains several [snip] Cheers, Vijay [snip] What happened to packages for RHEL/Centos 5? I have the (probably unusual--added gluster to existing servers) setup of running a replica 3 cluster where two nodes run on Centos 6 and one is still on Centos 5. This is a personal setup, and I have been using http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml as my repo. It has worked fine for a while, but this time the two Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind at 3.6.3. Packages for RHEL/CentOS-5 are not available yet. These will follow later. Thare are some changes needed to be able to build the packages on EL5. Because we are currently stabilizing our CI/regression tests, we do not merge any other changes. Until we provide packages in our repository, you could apply patch http://review.gluster.org/10803 yourself and build the EL5 version. I expect that we will do a release in 2-3 weeks which will have EL5 RPMs too. I have no idea about the problem below, it sounds like something the GlusterD developers could help with. Niels Command 'gluster volume status' on the C5 machine makes everything look fine: Status of volume: ISO2 Gluster process PortOnline Pid -- Brick 10.x.x.2:/bricks/01/iso249162 Y 4679 Brick 10.x.x.4:/bricks/01/iso249183 Y 6447 Brick 10.x.x.9:/bricks/01/iso249169 Y 1985 But the same command on either of the C6 machines shows the C5 machine (10.x.x.2) missing in action (though it does recognize that there are NFS and heal daemons there): Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid -- Brick 10.41.65.4:/bricks/01/iso249183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso249169 0 Y 1985 NFS Server on localhost 2049 0 Y 2279 Self-heal Daemon on localhost N/A N/AY 2754 NFS Server on 10.41.65.22049 0 Y 4757 Self-heal Daemon on 10.41.65.2 N/A N/AY 4764 NFS Server on 10.41.65.42049 0 Y 6543 Self-heal Daemon on 10.41.65.4 N/A N/AY 6551 So, is this just an oversight (I hope), or has support for C5 been dropped? If support for C5 is gone, how do I downgrade my Centos6 machines back to 3.6.x? (I know how to change the repo, but the actual sequence of yum commands and gluster commands is unknown to me). Could you attach the glusterd log file of 10.x.x.2 machine attached as etc-glusterfs-glusterd.vol.log.newer.2, starting from last machine reboot and the node from where you triggered volume status. attached as etc-glusterfs-glusterd.vol.log.newer4 starting same time as .2 log Could you also share gluster volume info output of all the nodes? I have several volumes, so I chose the one that shows up first on the listings: *from 10.41.65.2:* [root@office2 /var/log/glusterfs]$ gluster volume info Volume Name: ISO2 Type: Replicate Volume ID: 090da4b3-c666-41fe-8283-2c029228b3f7 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.41.65.2:/bricks/01/iso2 Brick2: 10.41.65.4:/bricks/01/iso2 Brick3: 10.41.65.9:/bricks/01/iso2 [root@office2 /var/log/glusterfs]$ gluster volume status ISO2 Status of volume: ISO2 Gluster process PortOnline Pid -- Brick 10.41.65.2:/bricks/01/iso2 49162 Y 4463 Brick 10.41.65.4:/bricks/01/iso2 49183 Y 6447 Brick 10.41.65.9:/bricks/01/iso2 49169 Y 1985 NFS Server on localhost 2049Y 4536 Self-heal Daemon on localhost N/A Y 4543 NFS Server on 10.41.65.9 2049Y 2279 Self-heal Daemon on 10.41.65.9 N/A Y 2754 NFS Server on 10.41.65.4 2049Y 6543 Self-heal Da
Re: [Gluster-users] [Gluster-devel] Gluster 3.7.0 released
From: Niels de Vos Sent: Monday, May 25, 2015 4:44 PM On Mon, May 25, 2015 at 06:49:26PM +, Ted Miller wrote: > > > From: Humble Devassy Chirammal > Sent: Monday, May 18, 2015 9:37 AM > Hi All, > > GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are > available at download.gluster.org<http://download.gluster.org> [1]. > > [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/ > > --Humble > > > On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur > mailto:vbel...@redhat.com>> wrote: > > Hi All, > > I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 > contains several > > [snip] > > Cheers, > Vijay > > [snip] > > What happened to packages for RHEL/Centos 5? I have the (probably > unusual--added gluster to existing servers) setup of running a replica > 3 cluster where two nodes run on Centos 6 and one is still on Centos > 5. This is a personal setup, and I have been using > http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml > as my repo. It has worked fine for a while, but this time the two > Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind > at 3.6.3. Packages for RHEL/CentOS-5 are not available yet. These will follow later. Thare are some changes needed to be able to build the packages on EL5. Because we are currently stabilizing our CI/regression tests, we do not merge any other changes. Until we provide packages in our repository, you could apply patch http://review.gluster.org/10803 yourself and build the EL5 version. I expect that we will do a release in 2-3 weeks which will have EL5 RPMs too. I have no idea about the problem below, it sounds like something the GlusterD developers could help with. Niels > Command 'gluster volume status' on the C5 machine makes everything > look fine: > > Status of volume: ISO2 > Gluster process PortOnline Pid > -- > Brick 10.x.x.2:/bricks/01/iso249162 Y 4679 > Brick 10.x.x.4:/bricks/01/iso249183 Y 6447 > Brick 10.x.x.9:/bricks/01/iso249169 Y 1985 > > But the same command on either of the C6 machines shows the C5 machine > (10.x.x.2) missing in action (though it does recognize that there are > NFS and heal daemons there): > > Status of volume: ISO2 > Gluster process TCP Port RDMA Port Online Pid > -- > Brick 10.41.65.4:/bricks/01/iso249183 0 Y 6447 > Brick 10.41.65.9:/bricks/01/iso249169 0 Y 1985 > NFS Server on localhost 2049 0 Y 2279 > Self-heal Daemon on localhost N/A N/AY 2754 > NFS Server on 10.41.65.22049 0 Y 4757 > Self-heal Daemon on 10.41.65.2 N/A N/AY 4764 > NFS Server on 10.41.65.42049 0 Y 6543 > Self-heal Daemon on 10.41.65.4 N/A N/AY 6551 > > So, is this just an oversight (I hope), or has support for C5 been dropped? > If support for C5 is gone, how do I downgrade my Centos6 machines back > to 3.6.x? (I know how to change the repo, but the actual sequence of > yum commands and gluster commands is unknown to me). > > Ted Miller > Elkhart, IN, USA Thanks for the information. As long as I know it is coming, I can improvise and hang on. I am assuming that the problem with the .2 machine not being seen is a result of running a cluster with a version split. Ted Miller ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] Gluster 3.7.0 released
From: Humble Devassy Chirammal Sent: Monday, May 18, 2015 9:37 AM Hi All, GlusterFS 3.7.0 RPMs for RHEL, CentOS, Fedora and packages for Debian are available at download.gluster.org<http://download.gluster.org> [1]. [1] http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/ --Humble On Thu, May 14, 2015 at 2:49 PM, Vijay Bellur mailto:vbel...@redhat.com>> wrote: Hi All, I am happy to announce that Gluster 3.7.0 is now generally available. 3.7.0 contains several [snip] Cheers, Vijay [snip] What happened to packages for RHEL/Centos 5? I have the (probably unusual--added gluster to existing servers) setup of running a replica 3 cluster where two nodes run on Centos 6 and one is still on Centos 5. This is a personal setup, and I have been using http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-5/x86_64/repodata/repomod.xml as my repo. It has worked fine for a while, but this time the two Centos 6 nodes updated to 3.7, but the Centos 5 node got left behind at 3.6.3. Command 'gluster volume status' on the C5 machine makes everything look fine: Status of volume: ISO2 Gluster process PortOnline Pid -- Brick 10.x.x.2:/bricks/01/iso249162 Y 4679 Brick 10.x.x.4:/bricks/01/iso249183 Y 6447 Brick 10.x.x.9:/bricks/01/iso249169 Y 1985 But the same command on either of the C6 machines shows the C5 machine (10.x.x.2) missing in action (though it does recognize that there are NFS and heal daemons there): Status of volume: ISO2 Gluster process TCP Port RDMA Port Online Pid -- Brick 10.41.65.4:/bricks/01/iso249183 0 Y 6447 Brick 10.41.65.9:/bricks/01/iso249169 0 Y 1985 NFS Server on localhost 2049 0 Y 2279 Self-heal Daemon on localhost N/A N/AY 2754 NFS Server on 10.41.65.22049 0 Y 4757 Self-heal Daemon on 10.41.65.2 N/A N/AY 4764 NFS Server on 10.41.65.42049 0 Y 6543 Self-heal Daemon on 10.41.65.4 N/A N/AY 6551 So, is this just an oversight (I hope), or has support for C5 been dropped? If support for C5 is gone, how do I downgrade my Centos6 machines back to 3.6.x? (I know how to change the repo, but the actual sequence of yum commands and gluster commands is unknown to me). Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ... i was able to produce a split brain...
On 2/3/2015 2:44 PM, Joe Julian wrote: On 02/03/2015 11:34 AM, Ted Miller wrote: On 2/3/2015 12:23 PM, Joe Julian wrote: That brought another thought to mind (have not had reason to try it): How does gluster cope if you go behind its back and rename a "rejected" file? For instance, in my example above, what if I go directly on the brick and rename the host-2 copy of the file to hair-pulling.txt-dud? The ideal scenario would seem to be that if user does a heal it would treat the copy as new file, see no dupe for hair-pulling.txt, and create a new dupe on host-2. Since hair-pulling.txt-dud is also a new file, a dupe would be created on host-1. User could then access files, verify correctness, and then delete hair-pulling.txt-dud. This should cause you to have two files with the same gfid. This will create the hardlink in .glusterfs again, and the heal will then re-create the .txt file also with that same gfid. Since both files will have the same gfid (stored in extended attributes) and be hard linked to the same file under .glusterfs you should then end up with both files being split-brain. Joe, I moved you comments up to be closest to the proposal they seem relevant to. * A not-officially-sanctioned way that I dealt with a split-brain a few versions back: 1. decided I wanted to keep file on host-2 2. log onto host-2 3. cp /brick/data1/hair-pulling.txt /gluster/data1/hair-pulling.txt-dud 4. rm /brick/data1/hair-pulling.txt 5. follow some Joe Julian blog stuff to delete the "invisible fork" of file 6. gluster volume heal data1 all If you note, in the above scenario _I copied from the brick to the mounted gluster volume_. I believe that this forces the breaking of any linkage between the old file and the new one. Am I missing something there? Yep, I missed that. I seem to be suffering from split-brain, myself, today. Don't sweat it, your blog posts have dug me out of more than one hole. Ted Miller Elkhart, IN, USA . // ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 答复: Re: replica position
On 1/29/2015 8:13 PM, yang.bi...@zte.com.cn wrote: example: brick A ,brick B,brick C compose one volume When client(fuse) in brick A write data into volume,for some reason ,we want the data wrote into brick B or brick C. Are you considering a replicated volume? If so, the _client_ will write to all three volumes at the same time. If you are looking at a distributed volume, then each file ends up on only one node, but file writing is distributed across different nodes. Ted Miller Elkhart, IN, USA 发件人: Pranith Kumar Karampuri 收件人: yang.bi...@zte.com.cn, gluster-users@gluster.org, 日期: 2015/01/28 16:40 主题: Re: [Gluster-users] replica position - On 01/27/2015 02:43 PM, _yang.bi...@zte.com.cn_ <mailto:yang.bi...@zte.com.cn>wrote: Hi, Can the replica position be configured ? Could you give an example? Pranith Regards. Bin.Yang ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately. ___ Gluster-users mailing list _Gluster-users@gluster.org_ <mailto:Gluster-users@gluster.org> _http://www.gluster.org/mailman/listinfo/gluster-users_ ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -- *Ted Miller*, Design Engineer *SonSet Solutions* (formerly HCJB Global Technology Center) my desk +1 574.970.4272 receptionist +1 574.972.4252 http://sonsetsolutions.org /Technology for abundant life!/ ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ... i was able to produce a split brain...
On 2/3/2015 12:23 PM, Joe Julian wrote: That brought another thought to mind (have not had reason to try it): How does gluster cope if you go behind its back and rename a "rejected" file? For instance, in my example above, what if I go directly on the brick and rename the host-2 copy of the file to hair-pulling.txt-dud? The ideal scenario would seem to be that if user does a heal it would treat the copy as new file, see no dupe for hair-pulling.txt, and create a new dupe on host-2. Since hair-pulling.txt-dud is also a new file, a dupe would be created on host-1. User could then access files, verify correctness, and then delete hair-pulling.txt-dud. This should cause you to have two files with the same gfid. This will create the hardlink in .glusterfs again, and the heal will then re-create the .txt file also with that same gfid. Since both files will have the same gfid (stored in extended attributes) and be hard linked to the same file under .glusterfs you should then end up with both files being split-brain. Joe, I moved you comments up to be closest to the proposal they seem relevant to. * A not-officially-sanctioned way that I dealt with a split-brain a few versions back: 1. decided I wanted to keep file on host-2 2. log onto host-2 3. cp /brick/data1/hair-pulling.txt /gluster/data1/hair-pulling.txt-dud 4. rm /brick/data1/hair-pulling.txt 5. follow some Joe Julian blog stuff to delete the "invisible fork" of file 6. gluster volume heal data1 all If you note, in the above scenario I copied from the brick to the mounted gluster volume. I believe that this forces the breaking of any linkage between the old file and the new one. Am I missing something there? Ted Miller Elkhart, IN ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] ... i was able to produce a split brain...
On 1/31/2015 12:47 AM, Pranith Kumar Karampuri wrote: On 01/30/2015 06:28 PM, Jeff Darcy wrote: Pranith and I had a discussion regarding this issue and here is what we have in our mind right now. We plan to provide the user commands to execute from mount so that he can access the files in split-brain. This way he can choose which copy is to be used as source. The user will have to perform a set of getfattrs and setfattrs (on virtual xattrs) to decide which child to choose as source and inform AFR with his decision. A) To know the split-brain status : getfattr -n trusted.afr.split-brain-status This will provide user with the following details - 1) Whether the file is in metadata split-brain 2) Whether the file is in data split-brain It will also list the name of afr-children to choose from. Something like : Option0: client-0 Option1: client-1 We also tell the user what the user could do to view metadata/data info; like stat to get metadata etc. B) Now the user has to choose one of the options (client-x/client-y..) to inspect the files. e.g., setfattr -n trusted.afr.split-brain-choice -v client-0 We save the read-child info in inode-ctx in order to provide the user access to the file in split-brain from that child. Once the user inspects the file, he proceeds to do the same from the other child of replica pair and makes an informed decision. C) Once the above steps are done, AFR is to be informed with the final choice for source. This is achieved by - (say the fresh copy is in client-0) e.g., setfattr -n trusted.afr.split-brain-heal-finalize -v client-0 This child will be chosen as source and split-brain resolution will be done. May I suggest another possible way to get around the difficulty in determining which of the files is the one to keep? What if each of the files were to be renamed by appending the name of the brick-host that it lives on? For example, in a replica 2 system: brick-1: data1 host-1: host1 brick-2: date1 host-2: host2 file name: hair-pulling.txt after running script/command to resolve split-brain, file system would have two files: hair-pulling.txt__host-1_data1 hair-pulling.txt__host-2_data1 the user would then delete the unwanted file and rename the wanted file back to hair-pulling.txt. The only problem would come with a very large file with a large number of replicas (like the replica 5 system I am working with). You might run out of space for all the copies. Otherwise, this seems (to me) to present a user-friendly way to do this. If the user has doubts (and disk space), user can choose to keep the rejected file around for a while, "just in case" it happens to have something useful in it that is missing from the "accepted" file. That brought another thought to mind (have not had reason to try it): How does gluster cope if you go behind its back and rename a "rejected" file? For instance, in my example above, what if I go directly on the brick and rename the host-2 copy of the file to hair-pulling.txt-dud? The ideal scenario would seem to be that if user does a heal it would treat the copy as new file, see no dupe for hair-pulling.txt, and create a new dupe on host-2. Since hair-pulling.txt-dud is also a new file, a dupe would be created on host-1. User could then access files, verify correctness, and then delete hair-pulling.txt-dud. * A not-officially-sanctioned way that I dealt with a split-brain a few versions back: 1. decided I wanted to keep file on host-2 2. log onto host-2 3. cp /brick/data1/hair-pulling.txt /gluster/data1/hair-pulling.txt-dud 4. rm /brick/data1/hair-pulling.txt 5. follow some Joe Julian blog stuff to delete the "invisible fork" of file 6. gluster volume heal data1 all I believe that this did work for me at that time. I have not had to do it on a recent gluster version. Ted Miller Elkhart, IN ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] how to restrict client connection to server to only one IP address
On 10/16/2014 2:48 PM, Łukasz Zygmański wrote: Hello, I am new to this list and new to GlusterFS, so I would be grateful if you could help me. I am trying to do this setup: client1(10.75.2.45) | | MTU 1500 V (10.75.2.41) gluster1gluster2 (10.75.2.43) ---> (10.75.2.44) <--- MTU 9000 In words, I have two glusterfs servers (in replication): gluster1 and gluster2 and a glusterfs client client1. The gluster1 has two network interfaces: 10.75.2.41 and 10.75.2.43. I would like gluster1 to communicate with gluster2 using jumbo frames and connection would be between interfaces 10.75.2.43 and 10.75.2.44. Since the client1 can only use default packet size (MTU 1500) I would like it to connect with gluster1 using only other network interface: 10.75.2.41. Is it possible? At the moment on gluster1 I have: eno16780032: flags=4163 mtu 1500 inet 10.75.2.43 netmask 255.255.255.0 broadcast 10.75.2.255 eno33559296: flags=4163 mtu 1500 inet 10.75.2.41 netmask 255.255.255.0 broadcast 10.75.2.255 and when I mount from client1 using: mount -t glusterfs 10.75.2.41:/vol1 /mnt/glusterfs it still uses connection to 10.75.2.43: # netstat -natup | egrep '(2.41|2.43)' tcp0 0 10.75.2.45:1020 10.75.2.43:49152 ESTABLISHED 10856/glusterfs tcp0 0 10.75.2.45:1022 10.75.2.41:24007 ESTABLISHED 10856/glusterfs Is there a way to restrict communication from client1 to gluster1 using only one IP address: 10.75.2.41? Any help would be much appreciated. Best regards Lukasz PS GlusterFS version on client: glusterfs-3.5.2-1.el7.x86_64 glusterfs-fuse-3.5.2-1.el7.x86_64 GlusterFS version on server: glusterfs-server-3.5.2-1.el7.x86_64 glusterfs-3.5.2-1.el7.x86_64 Since no one has answered this in a few days, I will try to do so, or at least start the process. You do not mention how the client connects. 1. If it is using gluster-fuse, what you are trying to do is futile, because the connections are not as you think. The data does not flow from client1 -> gluster1 -> gluster2. The way it really works is that client1 connects directly to both gluster1 and gluster2, and sends the data to both of them at the same time. The only time any volume of data transfers directly from gluster1 to gluster2 is during a heal operation. Unfortunately, gluster does not understand the concept of a separate "storage network" that the servers use to talk to each other. It only has one address, and that address is the one that the clients connect to. 2. If the client uses NFS, then you have something more like what you drew. The data passes client1 -> gluster1 via NFS, and then gluster1 -> gluster2. I am not using NFS, so I can't help you with if it is possible to have NFS on one network connection and gluster on a different connection, or what is required to accomplish this (if it can be done at all). Ted Miller ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] replaceing bricks from a cluster with replication
On 10/15/2014 3:46 PM, John G. Heim wrote: You don't say, so I will assume you have 2 bricks per machine, resulting in 2 names returned when you execute a gluster volume list command. I will assume that your volumes are named 'Dick' and 'Jane' and one brick per machine. Well, maybe I'm missing something but I have only one volume. Was I supposed to do it differently? You didn't miss anything at all :) It all depends on what you are trying to store on gluster, and how that storage needs (or doesn't need) to be separated into different volumes. If your data can all live happily together in one volume, that is just fine. Not everyone has storage requirements that are that simple. When they want to do something like this, they have to repeat for each separate volume. In my explanation, you just can ignore step 4! However, before you try my way, Joe Julian posted what looks like a simpler way to accomplish the same thing. You still have to work brick by brick, but he does the add and remove a brick in one step instead of my two. Also, he (very wisely) includes the warning about waiting until each brick is fully healed before going on to the next one. Depending on the number and size of your files, the heal may take quite a while. The heal may also interfere with normal access to the stored information, because it can completely tie up your network connection shoving data across it. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] replaceing bricks from a cluster with replication
On 10/9/2014 4:30 PM, John G. Heim wrote: How can I replace machines in a gluster cluster with triple replication? I've had crashes and I've replaces single machines before but always with one with the same name. Now I need to take machine1, machine2, and machine3 out of the cluster and replace it with machineA, machineB, and machineC. Gluster will not let me detach machine1 all by itself. I have to detach machine1, machine2, and machine3 all at the same time. But is that going to cause data loss? It seems to me that detaching all 3 machines in a brick on a triple replicated cluster would be like removing a single brick from a non-replicated cluster. And you can lose data in that case, right? I am just a user with a replica = 3 cluster running, but I will try to outline how I would to it. If I goof something up (or explain it badly) I hope that others will jump in and correct me. You don't say, so I will assume you have 2 bricks per machine, resulting in 2 names returned when you execute a gluster volume list command. I will assume that your volumes are named 'Dick' and 'Jane'. The only way I know how to do what you want to do is a rather tedious process, but here it is. I will not try to get all the commands word perfect, more like give you "pseudocode". The basic premise is that you cannot add "machines", you can only add and subtract bricks from a volume. 1. Add machineA to the cluster as a peer. 2. Add brick "DickA" on machineA to volume "Dick", resulting in a "replica 4" volume. 3. Remove brick "Dick1" on machine1 from volume "Dick", putting you back at a "replica 3" volume. 4. Repeat steps 2 & 3 for "JaneA" and "Jane1". 5. At this point you no longer have any data on machine1, so remove machine1 from the cluster. Repeat steps 1-5, substituting machineB and machine2. Repeat steps 1-5, substituting machineC and machine3. At that point you should be migrated from machine1,2,3 to machineA,B,C. Is that clear as mud? It is probably more tedious than what you wanted to do (especially if you have 5 or more volumes), but it is the only way I know to do it. Have fun, Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Hosed installation
On 10/7/2014 1:56 PM, Ryan Nix wrote: Hello, I seem to have hosed my installation while trying to replace a failed brick. The instructions for replacing the brick with a different host name/IP on the Gluster site are no longer available so I used the instructions from the Redhat Storage class that I attended last week, which assumed the replacement had the same host name. http://community.gluster.org/q/a-replica-node-has-failed-completely-and-must-be-replaced-with-new-empty-hardware-how-do-i-add-the-new-hardware-and-bricks-back-into-the-replica-pair-and-begin-the-healing-process/ It seems the working brick (I had two servers with simple replication only) will not release the DNS entry of the failed brick. Is there any way to simply reset Gluster completely? The simple way to "reset gluster completely" would be to delete the volume and start over. Sometimes this is the quickest way, especially if you only have one or two volumes. If nothing has changed, deleting the volume will not affect the data on the brick. You can either: Find and follow the instructions to delete the "markers" that glusterfs puts on the brick, in which case the create process should be the same as any new volume creation. Otherwise, when you do the "volume create..." step, it will give you an error, something like 'brick already in use'. You used to be able to override that by adding --force to the command line. (Have not needed it lately, so don't know if it still works.) Hope this helps Ted Miller Elkhart, IN Just to confirm, if I delete the volume so I can start over, deleting the volume will not delete the data. Is this correct? Finally, once the volume is deleted, do I have to do what Joe Julian recommended here? http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/ Thanks for any insights. - Ryan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster pool help
On 9/9/2014 7:12 PM, Michael Bushey wrote: I'm currently using GlusterFS 3.5.2 on a pair of production servers to share uploaded files and it's been reliable and working well with just the two servers. I've done some local tests of trying to add and remove servers and the results have not been good. I'm starting to think I have the wrong idea of what GlusterFS does and I'm trying to stick the square peg in the round hole. Is there some config to say "number of replicas = number of servers"? My need is to have a copy of the data local on each server. If I have 5 servers, I want five copies of the data. Basically each server should have it's own copy like it's a local ext4/zfs file-system, except it will immediately sync files added or removed on the other servers. I'm not sure the what the gluster definition of brick is, but I believe I want the number of bricks to equal the number of servers (at least for this particular file-system). I've played around a bit and lost my data every time I've tried to add or remove a node. There is plenty of documentation, but it all says the same thing and doesn't answer the basics. From `gluster volume info`: Number of Bricks: 1 x 2 = 2 I'm assuming the middle 2 is the number of bricks. I have no clue why we're multiplying by 1 to get itself. http://gluster.org/community/documentation/index.php/Gluster_3.2:_Displaying_Volume_Information - does not show this magic multiplication. The page for 3.3 does not exist. We're cloud based and we want to be able to add and remove servers on the fly. If for example I want to scale from 5 to 7 servers, it looks like I need to create a new storage pool with 7 replicas. Would it be the most logical to name my brick with the number of replicas? For example server1 to server5 already have /var/gluster/files5. I could then create /var/gluster/files7 on server1 to server7, create the pool with 7 replicas, then pick a machine like server1 to copy the data from /var/gluster/files5 to /var/gluster/files7, then destroy /var/gluster/files5. This seems extremely convoluted but it does not appear possible to expand a storage pool and expand the number of replicants with it. Thanks in advance for help and your time to read this. Michael I have done this a few times, but it is not real recent, so I will try to explain in a way that is both correct and understandable. 1. As to the question about the "`gluster volume info`: Number of Bricks: 1 x 2 = 2" The first number is the number of bricks that the data is distributed across. If that number is 3, each of three bricks has 1/3 of the data. This is the "Distribute" mode of operating gluster. From what you describe, you are not interested in that. You are interested in the "Replicate" mode. The numbering was adopted because gluster makes it possible to use various combinations of Distribute and Replicate at the same time, and this numbering describes the arrangement very succinctly. 2. As to how to you add a brick to your volume, so you can add a server to your system. You use the 'gluster volume add-brick' command. You are telling gluster: You are currently a two-brick system and I want to make you a 3-brick system (each brick being on a different server) by adding 'server3'. Typing 'gluster volume add-brick help' returns gluster volume add-brick [ ] ... [force] Assuming that the brick is ready and freshly formatted with xfs, the command ends up looking something like: gluster volume add-brick upload replica 3 server3:/bricks/upload/upload This will create the gluster structure on the brick, and will start the process of replicating the gluster data onto the brick. The gluster mount should be usable at that point, pulling the data from other bricks if it has not been replicated onto the local brick yet. If this server has been in service before, it is probably best to reformat the brick before putting it into service. Otherwise gluster will see remnants of the past use on the brick, and not want to use it, assuming there may be valuable information already stored there. CAUTION: Unless they have fixed this issue, the replication process can bring a server to its knees until it is fully replicated. 3. To remove a server from your system, the process is basically reversed. If you want to go from 5 servers to 4, you issue the command: gluster volume remove-brick upload replica 4 server5:/bricks/upload/upload This tells gluster to drop the brick on server5 from the volume, and reassures gluster that you know that you will now have a replicated volume with 4 replicas on it. Hope this helps. Ted Miller Elkhart, IN P.S. I am not reading the list regularly, so if you need more info, best to copy me when sending to the list. Just saw your request and said "I'v
Re: [Gluster-users] nfs
You do not create anything on "one volume". That will not replicate properly. Mount the /gluster/ volume -- in a different place -- and create the volume on the /gluster/ volume. It will then replicate to both bricks. Line in /etc/fstab should look like: primary:str /mnt/glusterfs *glusterfs*default 0 0 or primary:/str /mnt/glusterfs *nfs* default0 0 When you mount the /gluster/ volume, it does not "replicate from one brick to another" it is created on both bricks at the same time". If you mounted the volume correctly, you can just look at the bricks on each server, and you can see what is replicated. The entire file structure will be there. Ted Miller Elkhart, IN, USA P.S. This is one of the most common mistakes made by newbies. Don't feel bad if you made it. If you did not, congratulate yourself. On 5/28/2014 5:13 AM, yalla.gnan.ku...@accenture.com wrote: Hi All, I got it. Its due to lack of hostnames in /etc/hosts. Now it is working. J I am facing another problem. I have used replicated volumes in the two node gluster. I have used these replicated volume in openstack. I have created file system on one volume. But how to verify that the same content in one node is replicated on the gluster disk of another node ? Thanks Kumar *From:*Gnan Kumar, Yalla *Sent:* Wednesday, May 28, 2014 2:27 PM *To:* gluster-users@gluster.org *Subject:* nfs Hi All, I have a two node gluster storage. The gluster version is 3.2.5. But while mounting from a remote server as nfs volume, it is failing. -- root@compute:~# showmount -e primary Export list for primary: /rpl * /str * /dst * root@compute:~# mount -vvv -o mountproto=tcp -t nfs primary:/str /mnt/glusterfs mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID:0 mount: eUID: 0 mount: spec: "primary:/str" mount: node: "/mnt/glusterfs" mount: types: "nfs" mount: opts: "mountproto=tcp" mount: external mount: argv[0] = "/sbin/mount.nfs" mount: external mount: argv[1] = "primary:/str" mount: external mount: argv[2] = "/mnt/glusterfs" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw,mountproto=tcp" mount.nfs: timeout set for Wed May 28 06:13:38 2014 mount.nfs: trying text-based options 'mountproto=tcp,addr=10.211.203.66,mountaddr=10.211.203.66' mount.nfs: prog 13, trying vers=3, prot=6 mount.nfs: trying 10.211.203.66 prog 13 vers 3 prot TCP port 38467 mount.nfs: prog 15, trying vers=3, prot=6 mount.nfs: trying 10.211.203.66 prog 15 vers 3 prot TCP port 38465 mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting primary:/str What is the reason ? Thanks Kumar - This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. __ www.accenture.com ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- "He is no fool who gives what he cannot keep, to gain what he cannot lose." - - Jim Elliot For more information about Jim Elliot and his unusual life, see http://www.christianliteratureandliving.com/march2003/carolyn.html. Ted Miller Design Engineer HCJB Global Technology Center, a ministry of Reach Beyond 2830 South 17th St Elkhart, IN 46517 574--970-4272 my desk 574--970-4252 receptionist ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] services needed for nfs [SOLVED]
On 5/22/2014 12:13 PM, Ted Miller wrote: I have a running replica 3 setup, running on Centos 6, fully updated. I was trying to remove some unneeded services on a node (and did not document what I stopped). That node is now showing that NFS is N/A on that node. Since I already had one node showing NFS as N/A due to a "native" NFS share that was already in place, that leaves me with only one operation NFS server. What services need to be running to make the gluster-nfs shares available? What services should be turned off (or other changes made), in order to not have conflict between gluster-nfs and "native" NFS? Ted Miller Elkhart, IN, USA In my case, it was the nfslock service that was missing. I suspect there are others needed, but nfslock is definitely needed. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Proposal for improvements for heal commands
On 5/8/2014 7:12 AM, Pranith Kumar Karampuri wrote: [snip] 3) "gluster volume heal info split-brain" will be re-implemented to print all the files that are in split-brain instead of the limited 1024 entries. - One constant complaint is that even after the file is fixed from split-brain, it may still show up in the previously cached output. In this implementation the goal is to remove all the caching and compute the results afresh. Pranith, I missed your three-day deadline, because I was on vacation/holiday. If not for this time, perhaps for the next iteration. If at all possible, can we have the gfids in the split-brain output replaced by file names? No matter how you go about fixing a split-brain, not knowing the file name when faced with a list of possibilities is an infuriating situation. You can't tell log files from databases, so you can't even do triage. Thanks for your work, Ted Miller Elkhart, IN ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] services needed for nfs
I have a running replica 3 setup, running on Centos 6, fully updated. I was trying to remove some unneeded services on a node (and did not document what I stopped). That node is now showing that NFS is N/A on that node. Since I already had one node showing NFS as N/A due to a "native" NFS share that was already in place, that leaves me with only one operation NFS server. What services need to be running to make the gluster-nfs shares available? What services should be turned off (or other changes made), in order to not have conflict between gluster-nfs and "native" NFS? Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Replace dead nodes/bricks
A few things are not clear to me. Comments inline below.
On 5/15/2014 5:19 AM, Lyle Plaatjes wrote:
I've just started at a new company and I came across this problem. They
have webservers peering using gluster 3.2.7.
I take it that the gluster storage is on the same machines as the web-server
software?
Was this a replica-4 setup, where there is a replicated brick on each of 4
nodes?
The problem comes in where they upgraded the VM's to newer versions of
Ubuntu. They didn't replace the bricks before decommissioning the other
nodes. Only one node is actually still running so that is 1 brick that
actually exists and is being replicated to the new hosts.
So there really is no true replication going on, just all files being served
from the one gluster brick that still works? (And if that brick dies, the
whole site disappears until restored from backup {if there is a backup})
Now when one of the hosts is rebooted then gluster doesn't mount the volume
because it's looking at the 3 dead peers and one that is still fine.
Are your new nodes (without bricks) peers in the gluster cluster?
Are your mounts of the form localhost: ?
What I need to do is replace the old dead peers with the new ones so that
the gluster volume will actually mount if a host is rebooted.
Based on my guesses as to what your setup is, I think this is what I would do.
* Get all web servers operating as peers in the trusted pool
o It is not clear whether the new web servers even have gluster installed
* change /etc/fstab so that mounts are of the form localhost:
o so that it doesn't matter what other node is up or down, as long as
the volume is active
o I don't' know what exact commands Ubuntu is using, but in Centos 6 I
use the "nofail" option in the fourth column of /etc/fstab (where
'default' is a common entry).
+ This allows the bootup to proceed, even though the volume may not
be mountable yet.
# During (Centos) bootup, mount gets a second (or third) chance
to mount things
o make sure that the last two columns in /etc/fstab are "0 0"
+ so that it doesn't try to do a filesystem check during bootup
* set up bricks on new servers
o if the machine names are the same as the old machines, use a
different path from the old brick path
+ to see old brick path, run "gluster volume info "
o put brick in /etc/fstab, so it gets mounted
* run "gluster volume add-brick replica :/" on each node
o this adds the new bricks to the volume
o you may need to wait until one brick has healed (synchronized) before
adding the next brick
+ even synching one brick can saturate a network link, and bring
things to their knees
o repeat until you have 4 bricks active
* run "gluster volume remove-brick replica :/" to remove "historic" bricks
o don't do the remove bricks before adding any bricks
+ taking a replicated volume down to one brick and then trying to
bring it back to two bricks can be problematic on some (maybe
all) versions of gluster
Remember, I am assuming:
* you have 4 web servers that should also all be gluster brick nodes
* you were running a replica 4 configuration
Ted Miller
Elkhart, IN, USA
I am not an official part of gluster, just another user who has added and
removed bricks a few times.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Regarding Replicated Volume
On 3/19/2014 3:06 PM, Cary Tsai wrote: Hi There: New to the GlusterFS and could not find the answer from the document so hope I can get the answer form the mailing list. Let's say we have two web servers: One in Seattle, WA and another one is in Chapel Hill, NC. So I create a 'replicated' volume which one brick in WA and another brick in NC. I assume the web server in both WA and NC can mount the 'replicated' volume. There are 2 HTTP/Get calls from CA and NY. We assume CA's HTTP/Get is sent to web server in WA and NY's HTTP/Get is sent to web server in NC. My question is does the web server in WA definitely gets the data from the brick in WA? If not, is any way to configure so the web server in WA definitely gets data from the brick in WA? The answer to your basic question is either "yes" or "they're working on it". I know a while back it was on the "to do" list, but I am not sure if the patch is done, and if so, has it made it into production code. But, the last I heard, yes, we were heading that direction. Contrary to what someone else said, no, your scenario is not the only one where this is desirable. In most any active situation using mostly-read files, reading from your local replicated disk is much faster, and also reduces network activity. The big make-or-break questions as to whether this will work for you are: * How much do you write? (more=more problem) * Are these files sort-of/almost WORM files (Write Once, Read Many). WORM is better, RAM is worse * Do both servers write? (Only one is better) * Do you modify files? (More modification = more headaches) * Do you replace/update files? (Yes = more grief) The critical issue is timing. Gluster has various operations where it has to communicate with all nodes, and the process cannot move forward until all nodes answer. Gluster is designed for all nodes to be connected by 1GB or faster networking, so your cross-continental link is outside the use-case the developers are using. This always applies to writes, i.e. when a write occurs, it has to finish on both servers, probably with several commands issued, and each time it cannot go on to the next step until the distant server finishes. There are certain read operations where gluster checks to make sure that things match between all servers. I hear reference to the "stat" call as being one that can be slow, but I can't say I fully understand what it does. I think I understand that an 'ls' command does not include the 'stat' call, but the 'ls -l' does include the 'stat' call, so a 'ls -l' command on a directory with hundreds or thousands of files can take MUCH longer than an 'ls' call to that same directory. IF your web site is doing read-only access to your file system, and it is not triggering any calls that make gluster do a difference check between your two servers, it might work. If 1. You do not require absolute real-time synchronization between the servers AND 2. You can do all the writes on one of the two servers then you should probably look at Geo-replication. Geo-replication is a one-way process, where all the changes happen on one end, and they are reflected on the other end. It is designed to handle slower network links, and allows you to keep the two sites in close-to real-time synchronization. How close to real time will depend on your server write load, and you would have to describe what you are doing and let some of the folks here give you their experience in similar situations. At least you are within the intended use-case, so the developers will be receptive to any problems you have, and they may get fixed. Another caution (based on painfully learned experience). If you decide to try a regular (not Geo-Replicated) system, I advise that you store your data on a third machine somewhere, ESPECIALLY if both machines are updating files at the same time. Otherwise, it seems that it is only a matter of time before you will be struggling with a split-brain situation. When you face your first split-brain, you will wish you had never run into one. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Bring up a brick after disk failure
On 3/19/2014 7:17 AM, teg...@renget.se wrote: Hi, One of my bricks suffered from complete raid failure, (3 disks on raid6). I created a new raid, and wanted to bring the brick back up in the volume. Did the following 1. Removed it with gluster volume remove-brick gluster s1:/mnt/raid6 start gluster volume remove-brick gluster s1:/mnt/raid6 commit 2. Detached with gluster peer detach s1 3. Probed anew: gluster peer probe s1 4. Tried to add with gluster volume add-brick gluster s1:/mnt/raid6 But last one fails with Stage failed on operation 'Volume Add brick' I'm using 3.4.2. Thanks, /jon WARNING: I'm not even sure if this is true, or if I changed something else, but I give you my experience to try. I was having trouble adding an additional brick to my replica array, and I was getting the same (or a similar) message. I then tried adding the brick from the command line on the machine where the brick is located. All of the sudden I began to get meaningful error messages. As far as I know, the only thing I changed was which computer I sshed into, but it made all the difference in the world. The error messages were quite specific, and easy to understand. I am curious if others have experienced the same thing. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] missing dependency on glusterfs-??.rpm
I actually copied it from an oVirt node, where they get it from. Content of the /etc/yum.repos.d/glusterfs-epel.repo file is: # Place this file in your /etc/yum.repos.d/ directory [glusterfs-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/$basearch/ enabled=1 skip_if_unavailable=1 gpgcheck=0 [glusterfs-noarch-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/noarch enabled=1 skip_if_unavailable=1 gpgcheck=0 [glusterfs-source-epel] name=GlusterFS is a clustered file-system capable of scaling to several petabytes. - Source baseurl=http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/SRPMS enabled=0 skip_if_unavailable=1 gpgcheck=0 One of the reasons I chose that file is because I was extending a replica 3 volume on the ovirt nodes to a 4th (non-ovirt) node. "yum list gluster*" now spits back # yum list gluster* Loaded plugins: fastestmirror, presto Loading mirror speeds from cached hostfile * base: centos.mirror.nac.net * extras: mirror.cs.uwp.edu * updates: mirror.cs.vt.edu Installed Packages glusterfs.x86_64 3.4.2-1.el6 @glusterfs-epel glusterfs-cli.x86_64 3.4.2-1.el6 @glusterfs-epel glusterfs-fuse.x86_64 3.4.2-1.el6 @glusterfs-epel glusterfs-libs.x86_64 3.4.2-1.el6 @glusterfs-epel glusterfs-server.x86_643.4.2-1.el6 @glusterfs-epel Available Packages glusterfs-api.x86_64 3.4.2-1.el6 glusterfs-epel glusterfs-api-devel.x86_64 3.4.2-1.el6 glusterfs-epel glusterfs-debuginfo.x86_64 3.4.2-1.el6 glusterfs-epel glusterfs-devel.x86_64 3.4.2-1.el6 glusterfs-epel glusterfs-geo-replication.x86_64 3.4.2-1.el6 glusterfs-epel glusterfs-rdma.x86_64 3.4.2-1.el6 glusterfs-epel glusterfs-resource-agents.noarch 3.4.2-1.el6 glusterfs-noarch-epel On 3/13/2014 7:44 PM, Carlos Capriotti wrote: What link did you use to fetch your .repo file ? On Thu, Mar 13, 2014 at 11:27 PM, Ted Miller <mailto:tmil...@hcjb.org>> wrote: Sequence of events: spun up new Centos 6.5 VM yum -y update yum install glusterfs this installed gluster 3.4.0 find out there is no gluster-server package in CentOS repos. added glusterfs-epel, glusterfs-noarch-epel yum -y install gluster-server service glusterd start [*FAILED*] yum list gluster* note that all gluster* packages were updated to 3.4.2 /except/ yum update glusterfs-libs yum update glusterfs-libs service glusterd start [*OK *] Ted Miller Elkhart, IN ___ Gluster-users mailing list Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> http://supercolony.gluster.org/mailman/listinfo/gluster-users -- "He is no fool who gives what he cannot keep, to gain what he cannot lose." - - Jim Elliot For more information about Jim Elliot and his unusual life, see http://www.christianliteratureandliving.com/march2003/carolyn.html. Ted Miller Design Engineer HCJB Global Technology Center, a ministry of Reach Beyond 2830 South 17th St Elkhart, IN 46517 574--970-4272 my desk 574--970-4252 receptionist ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] PLEASE READ ! We need your opinion. GSOC-2014 and the Gluster community
[snip] 2) We have a recurring issue with split-brain solution. There is an entry on trello asking/suggesting a mechanism that arbitrates this resolution automatically. I pretty much think this could come together with another solution that is file replication consistency check. Anything to improve split-brain resolution would get my vote. Ted Miller Elkhart, IN ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] missing dependency on glusterfs-??.rpm
Sequence of events: spun up new Centos 6.5 VM yum -y update yum install glusterfs this installed gluster 3.4.0 find out there is no gluster-server package in CentOS repos. added glusterfs-epel, glusterfs-noarch-epel yum -y install gluster-server service glusterd start [*FAILED*] yum list gluster* note that all gluster* packages were updated to 3.4.2 /except/ yum update glusterfs-libs yum update glusterfs-libs service glusterd start [* OK *] Ted Miller Elkhart, IN ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] One node goes offline, the other node loses its connection to its local Gluster volume
On 3/6/2014 7:48 PM, Greg Scott wrote: In your real-life concern, the interconnect would not interfere with the existence of either machines' ip address so after the ping-timeout, operations would resume in a split-brain configuration. As long as no changes were made to the same file on both volumes, when the connection is reestablished, the self-heal will do exactly what you expect. Except that's not what happens. If I ifdown that interconnect NIC, I should see the file system after 42 seconds, right? But I don't. Email butchers the output below, but what it shows is, I can look at my /firewall-scripts directory just fine when things are steady state. I ifdown the interconnect NIC, that directory goes away. I wait more than 2 minutes and it still doesn't come back. And then when I ifup the NIC, everything goes back to normal after a few seconds. [root@stylmark-fw1 ~]# ls /firewall-scripts allow-all etc initial_rc.firewall rcfirewall.conf var allow-all-with-nat failover-monitor.sh rc.firewall start-failover-monitor.sh [root@stylmark-fw1 ~]# date Thu Mar 6 18:39:42 CST 2014 [root@stylmark-fw1 ~]# ifdown enp5s4 [root@stylmark-fw1 ~]# ls /firewall-scripts ls: cannot access /firewall-scripts: Transport endpoint is not connected [root@stylmark-fw1 ~]# date Thu Mar 6 18:41:50 CST 2014 [root@stylmark-fw1 ~]# ls /firewall-scripts ls: cannot access /firewall-scripts: No such file or directory [root@stylmark-fw1 ~]# ifup enp5s4 [root@stylmark-fw1 ~]# ls /firewall-scripts ls: cannot access /firewall-scripts: No such file or directory [root@stylmark-fw1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/fedora-root 17G 2.3G 14G 14% / devtmpfs 989M 0 989M 0% /dev tmpfs996M 0 996M 0% /dev/shm tmpfs996M 524K 996M 1% /run tmpfs996M 0 996M 0% /sys/fs/cgroup tmpfs996M 0 996M 0% /tmp /dev/sda2477M 87M 362M 20% /boot /dev/sda1200M 9.6M 191M 5% /boot/efi /dev/mapper/fedora-gluster--fw1 9.8G 33M 9.8G 1% /gluster-fw1 192.168.253.1:/firewall-scripts 9.8G 33M 9.8G 1% /firewall-scripts [root@stylmark-fw1 ~]# ls /firewall-scripts allow-all etc initial_rc.firewall rcfirewall.conf var allow-all-with-nat failover-monitor.sh rc.firewall start-failover-monitor.sh [root@stylmark-fw1 ~]# You can avoid the split-brain using a couple of quorum techniques, the one that would seem to satisfy your requirements leaving your volume read-only during the duration of the outage. I like this idea - how do I do it? I don't see a follow-up here, so I will put in (only) my two cents worth. If I understand correctly, you get the read-only condition by using client-side quorum. The behavior you describe above sounds like that produced by server-side quorum -- the volume goes offline until a quorum is present. I have suffered through a couple of split-brain situations, and I agree that you do not want to run a two-node setup without quorum. You may have gotten an answer that I did not see, but even so, I'll leave this here for the next guy who has a question. Ted Miller Elkhart, IN - Greg ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] overzealous quota?
After concluding that: Replica 2 + quorum = Low Availability Replica 2 - quorum = split brain (learned the hard way) I switched to replica 3 for my oVirt 2 host setup. Since I have only 2 hosts under the control of oVirt (which wants a LOT of control), I ended up with Replica 3 + quorum Brick 1 on host 1 Brick 2 on host 2 Brick 3 on host 2 Not ideal, but at least there are 3 bricks to vote, and host 1 should be able to go down and still have a quorum. What I find is that the situation is the same as replica 2 + quorum -- both hosts must be up in order for the volume to be accessible. Symptom gluster volume info: reports volume is started. gluster volume status: reports 2 bricks, but no ports allocated. files on gluster mount inaccessible. I realize that this is a corner case, but is there some reason I can't have my quorum all on one host? My setup: oVirt on Centos 6.5, updated yesterday. RFE #1: an easy, positive way to find out if a volume is available. Currently the only way I know is to look at "volume info" to see if it is started, then at "volume status" to see if ports are assigned. RFE #2: A way to designate that a volume would become read-only on lack of quorum, rather than taking it offline. I am sure I am not the only one with an almost-WORM use case, where having the volume go read-only would allow operations could continue as normal, just no new content could be added. In my use case I will be serving media files in real time. Under the current setup, I am considering mounting a brick as a read-only source, because that would allow operation to continue when quorum is lost. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] could I disable updatedb in brick server?
On 1/26/2014 6:13 AM, Mingfan Lu wrote: we have lots of (really) files in our gluser brick servers and every day, we will generate lots, the number of files increase very quickly. could I disable updatedb in brick servers? if that, glusterfs servers will be impacted? I don't have the docs open now, but there is a setup file where you can tell updatedb what to index and what to not index. I altered my configuration to add some things that it ignores by default, but it should be just as simple to tell it not to index your brick(s). Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] intresting issue of replication and self-heal
On 1/21/2014 11:05 PM, Mingfan Lu wrote: I have a volume (distribute-replica (*3)), today i found an interesting problem node22 node23 and node24 are the replica-7 from client A but the annoying thing is when I create dir or write file from client to replica-7, date;dd if=/dev/zero of=49 bs=1MB count=120 Wed Jan 22 11:51:41 CST 2014 120+0 records in 120+0 records out 12000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s but I could only find node23 & node24 have the find --- node23,node24 --- /mnt/xfsd/test-volume/test/49 in clientA, I use find command I use another machine as client B, and mount the test volume (newly mounted) to run*find /mnt/xfsd/test-volume/test/49* from Client A, the three nodes have the file now. --- node22,node23.node24 --- /mnt/xfsd/test-volume/test/49 but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22 still have the file in brick. --- node22 --- /mnt/xfsd/test-volume/test/49 but if i delete the new created files from Client B ) my question is why node22 have no newly created/write dirs/files? I have to use find to trigger the self-heal to fix that? from ClientA's log, I find something like: I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7: no active sinks for performing self-heal on file /test/49 It is harmless for it is information level? I also see something like: [2014-01-19 10:23:48.422757] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-test-volume-replicate-7: Non Blocking entrylks failed for /test/video/2014/01. [2014-01-19 10:23:48.423042] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-test-volume-replicate-7: background entry self-heal failed on /test/video/2014/01 From the paths you are listing, it looks like you may be mounting the bricks, not the gluster volume. You MUST mount the gluster volume, not the bricks that make up the volume. In your example, the mount looks like it is mounting the xfs volume. Your mount command should be something like: mount :test volume /mount/gluster/test-volume If a brick is part of a gluster volume, the brick must NEVER be written to directly. Yes, what you write MAY eventually be duplicated over to the other nodes, but if and when that happens is unpredictable. It will give the unpredictable replication results that you are seeing. The best way to test is to run "mount". If the line where you are mounting the gluster volume doesn't say "glusterfs" on it, you have it wrong. Also, the line you use in /etc/fstab must say "glusterfs", not "xfs" or "ext4". If you are in doubt, include the output of "mount" in your next email to the list. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] engine-iso-uploader -- REST API not usable?
I ran into this problem when I tried to use engine-iso-uploader, but reading on the lists makes it sound like it may be a more general problem. There was a bug that caused this, but that was back in the ver. 3.0/3.1 days, and doesn't seem common since then. Back on Dec 24 I was able to upload an ISO file OK, so I am not sure what has changed since then. I am running a test setup, fully up to date: office2a host w/ glusterfs Centos 6 office4a host w/ glusterfs Centos 6 ov-eng01 engine on Centos 6 VM (not hosted on oVirt) office9 KVM host (not oVirt) for ov-eng01 whether I log in to ov-eng01 by ssh or execute the command from the console, I get: # engine-iso-uploader list -v Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): ERROR: Problem connecting to the REST API. Is the service available and does the CA certificate exist? checking on some things suggested on a thread about engine-iso-uploader back in March, I get: # ls -la /etc/pki/ovirt-engine/ca.pem -rw-r--r--. 1 root root 4569 Nov 10 15:13 /etc/pki/ovirt-engine/ca.pem # cat /var/log/ovirt-engine/ovirt-iso-uploader/ovirt-iso-uploader/20140117112938.log 2014-01-17 11:29:44::ERROR::engine-iso-uploader::512::root:: Problem connecting to the REST API. Is the service available and does the CA certificate exist? The thread back in March gave a work-around to upload ISO images directly, so I am not "blocked" from uploading images, but I would like to get things working "right", as I am afraid the problem will "turn around and bite me" down the road. Ted Miller ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] split-brain + what?
From: Ted & Jean Miller Sent: Monday, January 13, 2014 11:29 PM To: Ted Miller Subject: split-brain + what? I had a network failure yesterday, and besides some split-brain files, I am getting the following in my files. It appears that my two nodes won't stay connected because they can't agree on something, but I don't understand what they don't agree on, or what to do about it. This log segment begins just after one of the nodes was rebooted. +--+ [2014-01-14 01:20:59.410888] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-VM-client-0: changing port to 49154 (from 0) [2014-01-14 01:20:59.411008] W [socket.c:514:__socket_rwv] 0-VM-client-0: readv failed (No data available) [2014-01-14 01:20:59.418326] I [client-handshake.c:1659:select_server_supported_programs] 0-VM-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-01-14 01:20:59.418462] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-VM-client-1: changing port to 49156 (from 0) [2014-01-14 01:20:59.418512] W [socket.c:514:__socket_rwv] 0-VM-client-1: readv failed (No data available) [2014-01-14 01:20:59.422397] I [client-handshake.c:1456:client_setvolume_cbk] 0-VM-client-0: Connected to 10.41.65.4:49154, attached to remote volume '/bricks/01/B'. [2014-01-14 01:20:59.422420] I [client-handshake.c:1468:client_setvolume_cbk] 0-VM-client-0: Server and Client lk-version numbers are not same, reopening the fds [2014-01-14 01:20:59.422481] I [afr-common.c:3698:afr_notify] 0-VM-replicate-0: Subvolume 'VM-client-0' came back up; going online. [2014-01-14 01:20:59.422617] I [client-handshake.c:450:client_set_lk_version_cbk] 0-VM-client-0: Server lk version = 1 [2014-01-14 01:20:59.422706] I [client-handshake.c:1659:select_server_supported_programs] 0-VM-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-01-14 01:20:59.422899] I [client-handshake.c:1456:client_setvolume_cbk] 0-VM-client-1: Connected to 10.41.65.2:49156, attached to remote volume '/bricks/01/B'. [2014-01-14 01:20:59.422909] I [client-handshake.c:1468:client_setvolume_cbk] 0-VM-client-1: Server and Client lk-version numbers are not same, reopening the fds [2014-01-14 01:20:59.429930] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 0 [2014-01-14 01:20:59.430077] I [client-handshake.c:450:client_set_lk_version_cbk] 0-VM-client-1: Server lk version = 1 [2014-01-14 01:20:59.431195] I [fuse-bridge.c:3724:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 [2014-01-14 01:20:59.432065] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-VM-replicate-0: added root inode [2014-01-14 01:20:59.432434] I [afr-common.c:2120:afr_discovery_cbk] 0-VM-replicate-0: selecting local read_child VM-client-0 [2014-01-14 03:00:47.506676] W [socket.c:514:__socket_rwv] 0-glusterfs: readv failed (No data available) [2014-01-14 03:00:47.506693] W [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (No data available), peer (10.41.65.4:24007) [2014-01-14 03:00:57.793050] I [glusterfsd-mgmt.c:1584:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2014-01-14 03:14:05.976531] W [socket.c:514:__socket_rwv] 0-VM-client-1: readv failed (Connection timed out) [2014-01-14 03:14:05.976563] W [socket.c:1962:__socket_proto_state_machine] 0-VM-client-1: reading from socket failed. Error (Connection timed out), peer (10.41.65.2:49156) [2014-01-14 03:14:05.976597] I [client.c:2097:client_rpc_notify] 0-VM-client-1: disconnected [2014-01-14 03:14:47.855141] E [client-handshake.c:1742:client_query_portmap_cbk] 0-VM-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2014-01-14 03:14:47.855192] W [socket.c:514:__socket_rwv] 0-VM-client-1: readv failed (No data available) [2014-01-14 03:14:47.855214] I [client.c:2097:client_rpc_notify] 0-VM-client-1: disconnected [2014-01-14 03:14:49.860364] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-VM-client-1: changing port to 49156 (from 0) [2014-01-14 03:14:49.860396] W [socket.c:514:__socket_rwv] 0-VM-client-1: readv failed (No data available) [2014-01-14 03:14:49.864065] I [client-handshake.c:1659:select_server_supported_programs] 0-VM-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-01-14 03:14:49.864325] I [client-handshake.c:1456:client_setvolume_cbk] 0-VM-client-1: Connected to 10.41.65.2:49156, attached to remote volume '/bricks/01/B'. [2014-01-14 03:14:49.864349] I [client-handshake.c:1468:client_setvolume_cbk] 0-VM-client-1: Server and Client lk-version numbers are not same, reopening the fds [2014-01-14 03:14:49.864527] I [client-handshake.c:450:client_set_lk_version_
Re: [Gluster-users] standby-server
-Ursprüngliche Nachricht- Von: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] Im Auftrag von Ted Miller Gesendet: Freitag, 16. August 2013 18:29 An: gluster-users@gluster.org Betreff: [Gluster-users] standby-server I am looking at glusterfs for a HA application w/o local tech support (don't ask, third-world country, techs are hard to find). My current plan is to do a replica-4 + hot spare server. Of the four in-use bricks, two will be on "servers" and the other two will be on a "client" machine and a "hot-backup" client machine. No striping, all content on each local machine, each machine using its own disk for all reading. Part of my plan is to have a cold-spare server in the rack, not powered on. (This server will also be a cold spare for another server). I am wondering if this would be a viable way to set up this configuration: Set up glusterfs as replica-5. 1. server1 2. server2 3. client 4. client-standby 5. server-spare Initialize and set up glusterfs with all 5 bricks in the system (no file content). Install system at client site, and test with all 5 bricks in system. Shut down spare server. Once a month, power up spare server, run full heal, shut down. Power up server-spare for any software updates. If server1 or server2 dies (or needs maintenance), tell them to power up server-spare, and let it heal. It seems to me that this would be easier than setting up a replica-4 system and then jumping through all the hoops to replace a server from scratch. Comments, reactions, pot-shots welcome. Ted Miller Ted Miller Design Engineer HCJB Global Technology Center Elkhart, IN 46517 On 8/19/2013 2:38 AM, Daniel Müller wrote: What ist he reason about a "spare server" ? With gluster just replicate to the real status all the time!? This is a hot, tropical location with terrible power, a lot of lightning storms and only part-time air-conditioning. Hardware failures are common. I am doing what I can to protect against the power problems, but it is still an environment that is very hard on equipment. This system will be business-critical--it is a radio station, and their music and other programs will be on the glusterfs, with downtime beyond seconds becoming dead air. If there is a catastrophic event, I want the spare server to have been off, and thus (most likely) not damaged. It can be brought on-line to restore dual server status. Or just do geo replication to a non gluster machine!? I plan on doing geo-replication, but restoring a multi-terabyte file system over a 4Mb link will take days. We would probably have to rsync the file system to portable drives and fly them over. Hope that helps the perspective a bit. Ted Miller --- EDV Daniel Müller Leitung EDV Tropenklinik Paul-Lechler-Krankenhaus Paul-Lechler-Str. 24 72076 Tübingen Tel.: 07071/206-463, Fax: 07071/206-499 eMail: muel...@tropenklinik.de Internet: www.tropenklinik.de --- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users Ted Miller Design Engineer HCJB Global Technology Center Elkhart, IN 46517 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] standby-server
I am looking at glusterfs for a HA application w/o local tech support (don't ask, third-world country, techs are hard to find). My current plan is to do a replica-4 + hot spare server. Of the four in-use bricks, two will be on "servers" and the other two will be on a "client" machine and a "hot-backup" client machine. No striping, all content on each local machine, each machine using its own disk for all reading. Part of my plan is to have a cold-spare server in the rack, not powered on. (This server will also be a cold spare for another server). I am wondering if this would be a viable way to set up this configuration: Set up glusterfs as replica-5. 1. server1 2. server2 3. client 4. client-standby 5. server-spare Initialize and set up glusterfs with all 5 bricks in the system (no file content). Install system at client site, and test with all 5 bricks in system. Shut down spare server. Once a month, power up spare server, run full heal, shut down. Power up server-spare for any software updates. If server1 or server2 dies (or needs maintenance), tell them to power up server-spare, and let it heal. It seems to me that this would be easier than setting up a replica-4 system and then jumping through all the hoops to replace a server from scratch. Comments, reactions, pot-shots welcome. Ted Miller -- "He is no fool who gives what he cannot keep, to gain what he cannot lose." - - Jim Elliot For more information about Jim Elliot and his unusual life, see http://www.christianliteratureandliving.com/march2003/carolyn.html. Ted Miller Design Engineer HCJB Global Technology Center 2830 South 17th St Elkhart, IN 46517 574--970-4272 my desk 574--970-4252 receptionist ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Some text do not display correctly in GlusterFS_Concepts page
The HTML contains the following at that spot: Error creating thumbnail: Unable to save thumbnail to destination Doesn't look like something we are supposed to see. Ted Miller On 8/6/2013 2:22 AM, Asias He wrote: Hello, In the Translator section of this link, there are some words in the background of the text. http://www.gluster.org/community/documentation/index.php/GlusterFS_Concepts -- "He is no fool who gives what he cannot keep, to gain what he cannot lose." - - Jim Elliot For more information about Jim Elliot and his unusual life, see http://www.christianliteratureandliving.com/march2003/carolyn.html. Ted Miller Design Engineer HCJB Global Technology Center 2830 South 17th St Elkhart, IN 46517 574--970-4272 my desk 574--970-4252 receptionist ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] nfs test dd cp tar
On 7/24/2013 4:10 PM, Heiko L. wrote:
Hallo
I tried glusterfs-3.3.2 dd cp tar.
- dd: ok
- cp: ok
- tar: fail (filesize=0)
Whats going wrong?
I am fairly new to digging this deep into Linux file systems, but I will put
my understanding into writing. I believe dd fails because glusterfs is
neither a file nor a block device. dd basically interacts with files. It
can also interact with disks and partitions because, under the /dev/
directory structure, entire file systems can be read and written to as though
they were one big file.
Gluster is different. It has no file system of its own--it borrows the file
system from whatever directory you tell it to use for its brick. When there
are redundant copies of a brick, those bricks are not all arranged the same,
if you dig down far enough. In fact, if I understand things correctly, on
one computer the files may be occupying an entire 500GB hard drive formatted
with XFS. On another computer, that same brick may occupy a /gluster
subdirectory on a 2TB drive which contains the entire root file system on an
ext4 drive. The way those files are stored on those two drives will be
different in many, many ways, but gluster is fine with that. However, that
makes it impossible to use dd to duplicate a "drive" like you can with other
file systems.
If you want to back up the file system, it will have to be by handling it as
a bunch of files, not as a block device. If you want to restore the files to
glusterfs again, you will need to make sure that your backup includes ALL the
extended attributes. I don't know exactly what commands it takes to do that,
but I do know that without the extended attributes glusterfs won't be able to
figure out if those files are the same as ones already on the system.
One other approach: if you are using identical partitions on all bricks to
hold the file system (ext4 or XFS) that glusterfs is using, you can do a dd
backup of the ext4 or XFS file system, and all the glusterfs information will
be there. If you do everything right, you can restore that backup back to a
hard drive, mount the hard drive in the brick's location, and glusterfs will
see it (assuming glusterfs configuration has not changed) and glusterfs will
welcome the new brick back into the volume group.
As I said, I am new to this, so I hope someone will correct any mistakes in
my explanation.
Ted Miller
regards Heiko
details
- test10 cp
root@z1-gluster:/mnt/gv3/test# sleep 10;date; cp -p /etc/profile .
Wed Jul 24 19:05:14 UTC 2013
root@z1-gluster:/mnt/gv3/test# sleep 10;date; cp -p /etc/profile test1
Wed Jul 24 19:05:24 UTC 2013
root@z1-gluster:/mnt/gv3/test# sleep 10;date; cp -p test1 test2
Wed Jul 24 19:05:35 UTC 2013
root@z1-gluster:/mnt/gv3/test# ls -l
total 4
-rw-r--r-- 1 root sys 1570 Jul 19 19:39 profile
-rw-r--r-- 1 root sys 1570 Jul 19 19:39 test1
-rw-r--r-- 1 root sys 1570 Jul 19 19:39 test2
drwxr-xr-x 14 root sys 14 Jul 20 11:05 var
root@z1-gluster:/mnt/gv3/test#
-> ok
---
- test11 dd
testfile=testfile
testdir=.
sz=1
time dd if=/dev/urandom bs=1024k count=$sz of=$testdir/${testfile}_${sz} 2>&1 |
grep -v rec
root@z1-gluster:/mnt/gv3/test# time dd if=/dev/urandom bs=1024k count=$sz
of=$testdir/${testfile}_${sz} 2>&1 | grep -v rec
real0m1.083s
user0m0.002s
sys 0m0.276s
root@z1-gluster:/mnt/gv3/test# ls -l ${testfile}*
-rw-r--r-- 1 root root 1048576 Jul 24 19:23 testfile_1
-rw-r--r-- 1 root root 10485760 Jul 24 19:24 testfile_10
-> ok
---
- test12 tar
root@z1-gluster:/mnt/gv3/test# src="profile test1 test2"
root@z1-gluster:/mnt/gv3/test# dst=test
profile
tar: profile: Cannot open: I/O error
test1
tar: test1: Cannot open: I/O error
test2
tar: test2: Cannot open: I/O error
tar: Exiting with failure status due to previous errors
root@z1-gluster:/mnt/gv3/test# ls -l $src $dst
-rw-r--r-- 1 root sys 1570 Jul 19 19:39 profile
-rw-r--r-- 1 root sys 1570 Jul 19 19:39 test1
-rw-r--r-- 1 root sys 1570 Jul 19 19:39 test2
test:
total 0
-rw--- 1 root root 0 Jul 24 19:55 profile
-rw--- 1 root root 0 Jul 24 19:55 test1
-rw--- 1 root root 0 Jul 24 19:55 test2
-> fail
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
"He is no fool who gives what he cannot keep, to gain what he cannot lose." - -
Jim Elliot
For more information about Jim Elliot and his unusual life, see
http://www.christianliteratureandliving.com/march2003/carolyn.html.
Ted Miller
Design Engineer
HCJB Global Technology Center
2830 South 17th St
[Gluster-users] rogue file
A multi-part question relating to a scenario where a rogue program/person writes a file into a brick through the underlying file system (e.g. XFS) without going through glusterfs? 1. What is gluster supposed to do when a non-gluster file (no gluster xattrs) appears on a gluster brick? 2. What does the current implementation actually do? 3. Does it matter whether it is or is not a replicated brick? 4. Does it matter whether it is a striped brick? Thinking about these scenarios after the file has been written: A. No one ever looks at the file B. A self-heal runs C. A directory read gets done on that sub-directory D. Someone actually tries to read the file (like if it was called readme.txt or some other common name) Don't have an example at hand, just wondering what would happen as I work on putting together a test bed that will be either replica 2 or replica 4, unstriped. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
