Re: [Gluster-users] Resync failure
On 28/09/2010 08:11, Marcus Bointon wrote: On 28 Sep 2010, at 06:30, Craig Carl wrote: The extended attributes on the files will be different between the two servers. Depending on the version of rsync you are running it may be reporting differences because of the attributes. Can you md5sum on both servers a couple of the files rsync is telling you are out-of-sync? If there isn't a difference in the md5sum values you are good to go. Otherwise please let us know. When I say they're out of sync I mean that there are files on one but not the other (both ways around, so both additions and deletions have not happened at some point) - I'm using cluster/replicate. Hi Marcus Can you confirm that you got into this situation by "fiddling" with the files outside of gluster? I hate reading reports like this on the list because it worries me that stuff can get out of sync, but in at least the majority of cases the "reason" for the lack of sync appears to be some variation of talking to the underlying volume directly rather than through the gluster mount point? Can you confirm your problem was traced to this? Cheers Ed W ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Resync failure
On 10/12/2010 06:22 AM, Marcus Bointon wrote: On 28 Sep 2010, at 09:11, Marcus Bointon wrote: When I say they're out of sync I mean that there are files on one but not the other (both ways around, so both additions and deletions have not happened at some point) - I'm using cluster/replicate. I'm using gluster purely for redundancy (images on a web server farm), not space or performance, and only have a very small volume of data. I could quite reasonably delete everything from the trailing node and copy back from the one that's ok. The main thing I'm not clear on is what (if any) data gluster stores apart from what's in the backing store, i.e. is it safe to stop servers, copy content around then restart them and expect them to be ok? Is there some secret data stash that will get confused if I do that? Just a little bump on this. Can I move stuff around while gluster isn't running and have it be OK when it starts back up? Or will I miss some other metadata store/cache? Can I force gluster to resync with the backing store? No, this is guaranteed to cause problems. Gluster stores information in the file's extended attributes (hence the requirement for filesystems that support extended attributes). If you don't have a lot of data, you might consider copying it off to non-Gluster storage, re-create your Gluster volume, and then copy the data back to the new Gluster volume. On a related topic, I'm also thinking about moving one of the nodes between servers since I need to take load off one of them - can I copy the backing store on to the new server and redeploy the new client configs and expect it to work? Marcus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Resync failure
On 28 Sep 2010, at 09:11, Marcus Bointon wrote: > When I say they're out of sync I mean that there are files on one but not the > other (both ways around, so both additions and deletions have not happened at > some point) - I'm using cluster/replicate. I'm using gluster purely for > redundancy (images on a web server farm), not space or performance, and only > have a very small volume of data. I could quite reasonably delete everything > from the trailing node and copy back from the one that's ok. > > The main thing I'm not clear on is what (if any) data gluster stores apart > from what's in the backing store, i.e. is it safe to stop servers, copy > content around then restart them and expect them to be ok? Is there some > secret data stash that will get confused if I do that? Just a little bump on this. Can I move stuff around while gluster isn't running and have it be OK when it starts back up? Or will I miss some other metadata store/cache? Can I force gluster to resync with the backing store? On a related topic, I'm also thinking about moving one of the nodes between servers since I need to take load off one of them - can I copy the backing store on to the new server and redeploy the new client configs and expect it to work? Marcus -- Marcus Bointon Synchromedia Limited: Creators of http://www.smartmessages.net/ UK resellers of i...@hand CRM solutions mar...@synchromedia.co.uk | http://www.synchromedia.co.uk/ ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Resync failure
On 28 Sep 2010, at 06:30, Craig Carl wrote: > The extended attributes on the files will be different between the two > servers. Depending on the version of rsync you are running it may be > reporting differences because of the attributes. Can you md5sum on both > servers a couple of the files rsync is telling you are out-of-sync? If there > isn't a difference in the md5sum values you are good to go. Otherwise please > let us know. When I say they're out of sync I mean that there are files on one but not the other (both ways around, so both additions and deletions have not happened at some point) - I'm using cluster/replicate. I'm using gluster purely for redundancy (images on a web server farm), not space or performance, and only have a very small volume of data. I could quite reasonably delete everything from the trailing node and copy back from the one that's ok. The main thing I'm not clear on is what (if any) data gluster stores apart from what's in the backing store, i.e. is it safe to stop servers, copy content around then restart them and expect them to be ok? Is there some secret data stash that will get confused if I do that? Marcus___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Resync failure
Marcus - The extended attributes on the files will be different between the two servers. Depending on the version of rsync you are running it may be reporting differences because of the attributes. Can you md5sum on both servers a couple of the files rsync is telling you are out-of-sync? If there isn't a difference in the md5sum values you are good to go. Otherwise please let us know. Thanks, Craig -- Craig Carl Sales Engineer; Gluster, Inc. Cell - ( 408) 829-9953 (California, USA) Office - ( 408) 770-1884 Gtalk - craig.c...@gmail.com Twitter - @gluster Installing Gluster Storage Platform, the movie! http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ From: "Marcus Bointon" To: "Gluster General Discussion List" Sent: Monday, September 27, 2010 5:34:46 AM Subject: [Gluster-users] Resync failure I'm running a very simple 2-way gluster 3.0.5 on ext3 with cluster/replicate (as generated by volgen) on 32-bit Ubuntu. I noticed that one of my gluster servers had been down for a while, so I fixed it and brought it back up. I wanted to be sure that all its content was in sync so I issued an ls -lR on the mounted volume, then used rsync to compare (but not touch) the backing file systems. There are hundreds of differences listed. What's the best way of manually repairing it, and why didn't it fix itself? Can I stop all servers, do an rsync and then restart? Marcus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Resync failure
I'm running a very simple 2-way gluster 3.0.5 on ext3 with cluster/replicate (as generated by volgen) on 32-bit Ubuntu. I noticed that one of my gluster servers had been down for a while, so I fixed it and brought it back up. I wanted to be sure that all its content was in sync so I issued an ls -lR on the mounted volume, then used rsync to compare (but not touch) the backing file systems. There are hundreds of differences listed. What's the best way of manually repairing it, and why didn't it fix itself? Can I stop all servers, do an rsync and then restart? Marcus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
