Hi, I will try to reproduce this in a vagrant-cluster environnement.
If it can help, here is the time line of the event. t0: 2 serveurs in replicate mode no issue t1: power down server1 due to hardware issue t2: server2 still continue to serve files through NFS and fuse, and still continue to being updated by automated build process / copy from other places t3: power up server1 which has been fixed, auto healing start for files, in the mean time we had a Jenkins job deploying files and remove/create symlinks to the proper target (through NFS and fuse). t4: Heal failed on some directory, in fact some symlinks in those directory has not been updated from those located on server2 and we could not access those directory anymore (I/O Error). t5: Removing the symlinks from server1 (directly on the brick), start a new replicate from server2, the symlink are now consistent between the two servers. We didn’t add/remove brick during this process. As I can see, the split-mount help to remove the bad files from bricks in a split-brain like event. (That what we have done by hand so far). Thanks On Apr 19, 2014, at 9:42 AM, Joe Julian <j...@julianfamily.org<mailto:j...@julianfamily.org>> wrote: What would really help is a clear list of steps to reproduce this issue. It sounds like a bug but I can't repro. In your questions you ask in relation to adding or removing bricks whether you can continue to read and write. My understanding is that you're not doing that (gluster volume (add|remove)-brick) but rather just shutting down. If my understanding is correct, then yes. You should be able to continue normal operation. Repairing this issue is the same as healing split-brain. The easiest way is to use splitmount[1] to delete one of them. [1] https://forge.gluster.org/splitmount On April 17, 2014 4:37:32 PM PDT, "PEPONNET, Cyril (Cyril)" <cyril.pepon...@alcatel-lucent.com<mailto:cyril.pepon...@alcatel-lucent.com>> wrote: Hi gluster people ! I would like some help regarding an issue we have with our early production glusterfs setup. Our Topology: 2 Bricks in Replicate mode: [root@myBrick1 /]# cat /etc/redhat-release CentOS release 6.5 (Final) [root@myBrick1 /]# glusterfs --version glusterfs 3.4.2 built on Jan 3 2014 12:38:05 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> [root@myBrick1 /]# gluster volume info Volume Name: myVol Type: Replicate Volume ID: 58f5d775-acb5-416d-bee6-5209f7b20363 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: myBrick1.company.lan:/export/raid/myVol Brick2: myBrick2.company.lan:/export/raid/myVol Options Reconfigured: nfs.enable-ino32: on The issue: We power down a brick (myBrick1) for hardware maintenance, when we power it up, issues starts with some files (symlinks in fact), auto healing seems not working fine for all the files… Let's take a look with one faulty symlink: Using fuse.glusterfs (sometimes it works sometimes not) [root@myBrick2 /]mount ... myBrick2.company.lan:/myVol on /images type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) ... [root@myBrick2 /]# stat /images/myProject1/2.1_stale/current File: `/images/myProject1/2.1_stale/current' -> `current-59a77422' Size: 16 Blocks: 0 IO Block: 131072 symbolic link Device: 13h/19d Inode: 11422905275486058235 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 499/ testlab) Gid: ( 499/ testlab) Access: 2014-04-17 14:05:54.488238322 -0700 Modify: 2014-04-16 19:46:05.033299589 -0700 Change: 2014-04-17 14:05:54.487238322 -0700 [root@myBrick2 /]# stat /images/myProject1/2.1_stale/current stat: cannot stat `/images/myProject1/2.1_stale/current': Input/output error I type the above commands with few seconds between them. Let's try with the other brick [root@myBrick1 ~]mount ... myBrick1.company.lan:/myVol on /images type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) ... [root@myBrick1 ~]# stat /images/myProject1/2.1_stale/current stat: cannot stat `/images/myProject1/2.1_stale/current': Input/output error With this one it always fail… (myBrick1 is the server we powered up after maintenance). Using nfs: It never works (tested with two bricks) [root@station-localdomain myProject1]# mount ... myBrick1:/myVol on /images type nfs (rw,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=14,retrans=2,sec=sys,mountaddr=10.0.0.57,mountvers=3,mountport=38465,mountproto=tcp,local_lock=none,addr=10.0.0.57) ... [root@station-localdomain myProject1]# ls 2.1_stale ls: cannot access 2.1_stale: Input/output error In both cases here are the logs: ==> /var/log/glusterfs/glustershd.log <== [2014-04-17 10:20:25.861003] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: <gfid:fcbbe770-6388-4d74-a78a-7939b17e36aa>: Performing conservative merge [2014-04-17 10:20:25.895143] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: <gfid:ae058719-61de-47de-82dc-6cb8a3d80afe>: Performing conservative merge [2014-04-17 10:20:25.949176] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: <gfid:868e3eb7-03e6-4b6b-a75a-16b31bdf8a10>: Performing conservative merge [2014-04-17 10:20:25.995289] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: <gfid:115efb83-2154-4f9d-8c70-a31f476db110>: Performing conservative merge [2014-04-17 10:20:26.013995] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: <gfid:0982330a-2e08-4b97-9ea5-cf991d295e41>: Performing conservative merge [2014-04-17 10:20:26.050693] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: <gfid:3a15b54b-b92c-4ed5-875e-1af0a3b94e0c>: Performing conservative merge ==> /var/log/glusterfs/usr-global.log <== [2014-04-17 10:20:38.281705] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: /images/myProject1/2.1_stale: Performing conservative merge [2014-04-17 10:20:38.286986] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 1 [2014-04-17 10:20:38.287030] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:38.287169] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 1 [2014-04-17 10:20:38.287202] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:38.287280] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:38.287308] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:38.287506] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 1 [2014-04-17 10:20:38.287538] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:38.311222] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 0 [2014-04-17 10:20:38.311277] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:38.311345] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 0 [2014-04-17 10:20:38.311385] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:38.311473] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 0 [2014-04-17 10:20:38.311502] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:38.332110] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:38.332149] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:38.332845] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-myVol-replicate-0: background entry self-heal failed on /images/myProject1/2.1_stale [2014-04-17 10:20:41.447911] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: /images/myProject1/2.1_stale: Performing conservative merge [2014-04-17 10:20:41.453950] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 1 [2014-04-17 10:20:41.453998] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:41.454135] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 1 [2014-04-17 10:20:41.454163] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:41.454237] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:41.454263] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:41.454385] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 1 [2014-04-17 10:20:41.454413] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:41.479015] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 0 [2014-04-17 10:20:41.479063] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:41.479149] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 0 [2014-04-17 10:20:41.479177] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:41.479252] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 0 [2014-04-17 10:20:41.479279] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:41.499291] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:41.499333] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:41.499995] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-myVol-replicate-0: background entry self-heal failed on /images/myProject1/2.1_stale [2014-04-17 10:20:43.149818] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: /images/myProject1/2.1_stale: Performing conservative merge [2014-04-17 10:20:43.155127] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 1 [2014-04-17 10:20:43.155185] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:43.155308] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 0 [2014-04-17 10:20:43.155346] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:43.155441] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 0 [2014-04-17 10:20:43.155477] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:43.155628] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 1 [2014-04-17 10:20:43.155660] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:43.180271] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 0 [2014-04-17 10:20:43.180324] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:43.180425] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 0 [2014-04-17 10:20:43.180455] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:43.180545] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 0 [2014-04-17 10:20:43.180578] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:43.201070] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:43.201112] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:43.201788] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-myVol-replicate-0: background entry self-heal failed on /images/myProject1/2.1_stale [2014-04-17 10:20:44.646242] I [afr-self-heal-entry.c:2253:afr_sh_entry_fix] 0-myVol-replicate-0: /images/myProject1/2.1_stale: Performing conservative merge [2014-04-17 10:20:44.652027] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 1 [2014-04-17 10:20:44.652072] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:44.652207] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 1 [2014-04-17 10:20:44.652239] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:44.652341] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:44.652372] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:44.652518] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 1 [2014-04-17 10:20:44.652550] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:44.676929] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_s: gfid differs on subvolume 0 [2014-04-17 10:20:44.676973] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_s [2014-04-17 10:20:44.677062] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_b: gfid differs on subvolume 0 [2014-04-17 10:20:44.677107] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_b [2014-04-17 10:20:44.677196] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/current: gfid differs on subvolume 0 [2014-04-17 10:20:44.677225] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/current [2014-04-17 10:20:44.698071] W [afr-common.c:1505:afr_conflicting_iattrs] 0-myVol-replicate-0: /images/myProject1/2.1_stale/latest_n: gfid differs on subvolume 1 [2014-04-17 10:20:44.698113] E [afr-self-heal-common.c:1433:afr_sh_common_lookup_cbk] 0-myVol-replicate-0: Conflicting entries for /images/myProject1/2.1_stale/latest_n [2014-04-17 10:20:44.698816] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-myVol-replicate-0: background entry self-heal failed on /images/myProject1/2.1_stale So let’s take a deeper look. Any split-brain ? [root@myBrick1 glusterfs]# gluster volume heal myVol info split-brain Gathering Heal info on volume myVol has been successful Brick myBrick1.company.lan:/export/raid/myVol Number of entries: 0 Brick myBrick2.company.lan:/export/raid/myVol Number of entries: 0 Nop Any heal-failed ? [root@myBrick1 glusterfs]# gluster volume heal myVol info heal-failed |wc -l 380 Plenty… how unique of them ? [root@myBrick1 glusterfs]# gluster volume heal myVol info heal-failed | cut -d " " -f 3 | sort -u | wc -l 18 Digging on it Here are failling entry from gluster volume myVol info heal-failed <gfid:0982330a-2e08-4b97-9ea5-cf991d295e41> <gfid:29337848-ffad-413b-91b1-7bd062b8c939> <gfid:3140c4f6-d95c-41bb-93c4-18a644497160> <gfid:5dd03e08-c9b6-4315-b6ae-efcb45558f18> <gfid:75db102b-99f4-4852-98b6-d43e39c3ccb6> <gfid:9c193529-75bf-4f81-bbd6-95a952d646dd> <gfid:a21c5f72-6d05-4c56-a34f-fcbef48374da> <gfid:f6660583-c8a7-4d4a-88d8-1138fc1030f5> /images/myProject3/2.1 /images/myProject3/2.1_stale /images/myProject2/2.1 /images/myProject2/2.1_stale /images/myProject1/2.1 /images/myProject1/2.1_stale Lets compare their xattr getfattr from file above : myBrick1: /export/raid/myVol/images/myProject3/2.1 -> trusted.gfid=0x2d18a6f72a894f20a260478b5a9602be /export/raid/myVol/images/myProject3/2.1_stale -> trusted.gfid=0x29337848ffad413b91b17bd062b8c939 /export/raid/myVol/images/myProject2/2.1 -> trusted.gfid=0x04cdfe8bb83b4b27b42153df913b5181 /export/raid/myVol/images/myProject2/2.1_stale -> trusted.gfid=0x5dd03e08c9b64315b6aeefcb45558f18 /export/raid/myVol/images/myProject1/2.1 -> trusted.gfid=0xca8fedea8ad64612a33db75ea1ca4421 /export/raid/myVol/images/myProject1/2.1_stale -> trusted.gfid=0xa21c5f726d054c56a34ffcbef48374da myBrick2: /export/raid/myVol/images/myProject3/2.1 -> trusted.gfid=0x2d18a6f72a894f20a260478b5a9602be /export/raid/myVol/images/myProject3/2.1_stale -> trusted.gfid=0x29337848ffad413b91b17bd062b8c939 /export/raid/myVol/images/myProject2/2.1 -> trusted.gfid=0x04cdfe8bb83b4b27b42153df913b5181 /export/raid/myVol/images/myProject2/2.1_stale -> trusted.gfid=0x5dd03e08c9b64315b6aeefcb45558f18 /export/raid/myVol/images/myProject1/2.1 -> trusted.gfid=0xca8fedea8ad64612a33db75ea1ca4421 /export/raid/myVol/images/myProject1/2.1_stale -> trusted.gfid=0xa21c5f726d054c56a34ffcbef48374da Damns they seems good, I made a cross check md5sum all files are the same on both bricks, Let chek the symlinks now myBrick1: /export/raid/myVol/images/myProject3/2.1_stale/current xattr=trusted.gfid=0x95b01ba94e0c482eacf51ebb20c1cba1 /export/raid/myVol/images/myProject3/2.1_stale/latest_b xattr=trusted.gfid=0x5c0165dfe5c84c7ea076731065292135 /export/raid/myVol/images/myProject3/2.1_stale/latest_n xattr=trusted.gfid=0xab4de5d630084cde891ac65a7904f6d0 /export/raid/myVol/images/myProject3/2.1_stale/latest_s xattr=trusted.gfid=0x4c38edac7d4c4e5e8ad8ff44a164f7b8 /export/raid/myVol/images/myProject2/2.1_stale/current xattr=trusted.gfid=0x946ce9e7224f4fc581d817a1ebcec087 /export/raid/myVol/images/myProject2/2.1_stale/latest_b xattr=trusted.gfid=0x8fb0020e97a54be0953e4786c6933f86 /export/raid/myVol/images/myProject2/2.1_stale/latest_n xattr=trusted.gfid=0x8de1551a69f244e3bb0a61cbaba57414 /export/raid/myVol/images/myProject2/2.1_stale/latest_s xattr=trusted.gfid=0xd11e42f54a2944a68ee7fd1f544539a9 /export/raid/myVol/images/myProject1/2.1_stale/current xattr=trusted.gfid=0x1ce5eac809694d83b983023efaea0f64 /export/raid/myVol/images/myProject1/2.1_stale/latest_b xattr=trusted.gfid=0xcc25cfdf98f749caaf259c76fe1b85b1 /export/raid/myVol/images/myProject1/2.1_stale/latest_n xattr=trusted.gfid=0x3889f78789a14b388e9fb5caa2231cc7 /export/raid/myVol/images/myProject1/2.1_stale/latest_s xattr=trusted.gfid=0x2f9901e45c5a47d282faaf65c675cf48 myBrick2: /export/raid/myVol/images/myProject3/2.1_stale/current xattr=trusted.gfid=0xb6d0a17f397b4922a0ac0e3d740ca8c7 /export/raid/myVol/images/myProject3/2.1_stale/latest_b xattr=trusted.gfid=0xc80750eca11b40f48feb99ea6cd07799 /export/raid/myVol/images/myProject3/2.1_stale/latest_n xattr=trusted.gfid=0x57bb90d64bd74c29ab30d844b33528b7 /export/raid/myVol/images/myProject3/2.1_stale/latest_s xattr=trusted.gfid=0x47f94619468a419d98011d8e67a43068 /export/raid/myVol/images/myProject2/2.1_stale/current xattr=trusted.gfid=0xcda7ba2331e6489f95c524a17ae179bf /export/raid/myVol/images/myProject2/2.1_stale/latest_b xattr=trusted.gfid=0xc36205fa8d2c49bda64001d667aab8a6 /export/raid/myVol/images/myProject2/2.1_stale/latest_n xattr=trusted.gfid=0xf71b8f6a4ff14e75951b47ae40817b70 /export/raid/myVol/images/myProject2/2.1_stale/latest_s xattr=trusted.gfid=0xcba23dbde7b14b63acc43d287a1b527e /export/raid/myVol/images/myProject1/2.1_stale/current xattr=trusted.gfid=0x82434b00c17d4d7d88cff71ba4d8c10d /export/raid/myVol/images/myProject1/2.1_stale/latest_b xattr=trusted.gfid=0xbba1934c4f7240bab705bd0548fbdc22 /export/raid/myVol/images/myProject1/2.1_stale/latest_n xattr=trusted.gfid=0x408ef2e2c1c5454fb0b920065eebab2f /export/raid/myVol/images/myProject1/2.1_stale/latest_s xattr=trusted.gfid=0x9fc2992ad16c4186a0d36ef93fea73f1 Damn, trusted.gfid is not consistent between the bricks… how does it append ? When we power up the myBrick1 we have some automated jobs writing on the other brick through nfs, and this job copying a bunch of file and update the symlinks above. So definitely there is an issue with replication… but only with symlinks ??? So here are the questions: 1) Can we still use glusters in read/write operation when adding a new or old brick ? (down for maintenance for example) ? That a key point in our deployment for scalability and flexibility 2) How can I recover / delete the conflicted grid files ? (as symlinks are the same in both side, only xattr differs). Thanks a lot for your help Cyril ________________________________ Gluster-users mailing list Gluster-users@gluster.org<mailto:Gluster-users@gluster.org> http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users