Re: [Gluster-users] intresting issue of replication and self-heal
On 1/21/2014 11:05 PM, Mingfan Lu wrote: I have a volume (distribute-replica (*3)), today i found an interesting problem node22 node23 and node24 are the replica-7 from client A but the annoying thing is when I create dir or write file from client to replica-7, date;dd if=/dev/zero of=49 bs=1MB count=120 Wed Jan 22 11:51:41 CST 2014 120+0 records in 120+0 records out 12000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s but I could only find node23 node24 have the find --- node23,node24 --- /mnt/xfsd/test-volume/test/49 in clientA, I use find command I use another machine as client B, and mount the test volume (newly mounted) to run*find /mnt/xfsd/test-volume/test/49* from Client A, the three nodes have the file now. --- node22,node23.node24 --- /mnt/xfsd/test-volume/test/49 but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22 still have the file in brick. --- node22 --- /mnt/xfsd/test-volume/test/49 but if i delete the new created files from Client B ) my question is why node22 have no newly created/write dirs/files? I have to use find to trigger the self-heal to fix that? from ClientA's log, I find something like: I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7: no active sinks for performing self-heal on file /test/49 It is harmless for it is information level? I also see something like: [2014-01-19 10:23:48.422757] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-test-volume-replicate-7: Non Blocking entrylks failed for /test/video/2014/01. [2014-01-19 10:23:48.423042] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-test-volume-replicate-7: background entry self-heal failed on /test/video/2014/01 From the paths you are listing, it looks like you may be mounting the bricks, not the gluster volume. You MUST mount the gluster volume, not the bricks that make up the volume. In your example, the mount looks like it is mounting the xfs volume. Your mount command should be something like: mount host name:test volume /mount/gluster/test-volume If a brick is part of a gluster volume, the brick must NEVER be written to directly. Yes, what you write MAY eventually be duplicated over to the other nodes, but if and when that happens is unpredictable. It will give the unpredictable replication results that you are seeing. The best way to test is to run mount. If the line where you are mounting the gluster volume doesn't say glusterfs on it, you have it wrong. Also, the line you use in /etc/fstab must say glusterfs, not xfs or ext4. If you are in doubt, include the output of mount in your next email to the list. Ted Miller Elkhart, IN, USA ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] intresting issue of replication and self-heal
more hints: I found node23 node24 have many files in .glusterfs/indices/xattrop there should be some problem? who could give some suggestions to resolve it? On Thu, Jan 23, 2014 at 5:04 PM, Mingfan Lu mingfan...@gmail.com wrote: I profiled node22, I found that most latency comes from setxattr, where node23 node22 comes from lookup and locks. any one could help? %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us2437540 FORGET 0.00 0.00 us 0.00 us 0.00 us 252684 RELEASE 0.00 0.00 us 0.00 us 0.00 us2226292 RELEASEDIR 0.00 38.00 us 37.00 us 40.00 us 4 FGETXATTR 0.00 66.16 us 15.00 us 13139.00 us596 GETXATTR 0.00 239.14 us 58.00 us 126477.00 us 1967 LINK 0.00 51.85 us 14.00 us8298.00 us 19045 STAT 0.00 165.50 us 9.00 us 212057.00 us 20544 READDIR 0.001827.92 us 184.00 us 150298.00 us 2084 RENAME 0.00 49.14 us 12.00 us5908.00 us 189019 STATFS 0.00 84.63 us 14.00 us 96016.00 us 163405 READ 0.00 29968.76 us 156.00 us 1073902.00 us 3115 CREATE 0.001340.25 us 6.00 us 7415357.00 us 248141 FLUSH 0.001616.76 us 32.00 us 13865122.00 us 229190 FTRUNCATE 0.011807.58 us 19.00 us 55480776.00 us 249569OPEN 0.011875.11 us 10.00 us 8842171.00 us 465197 FSTAT 0.05 393296.28 us 52.00 us 56856581.00 us 9057 UNLINK 0.07 32291.01 us 192.00 us 9638107.00 us 156081 RMDIR 0.08 18339.18 us 140.00 us 5313885.00 us 337862 MKNOD 0.092904.39 us 18.00 us 51724741.00 us2226290 OPENDIR 0.154708.15 us 27.00 us 55115760.00 us2334864 SETXATTR 0.188965.91 us 68.00 us 26465968.00 us1513280 FXATTROP 0.213465.29 us 74.00 us 58580783.00 us4506602 XATTROP 0.284801.16 us 44.00 us 49643138.00 us4436847 READDIRP 0.375935.92 us 7.00 us 56449083.00 us4611760 ENTRYLK 1.024226.58 us 33.00 us 63494729.00 us 18092335 WRITE 1.502734.50 us 6.00 us 185109908.00 us 40971541 INODELK 4.75 348602.30 us 5.00 us 2185602946.00 us1019332 FINODELK * 14.98 33957.49 us 14.00 us 59261447.00 us 32998211 LOOKUP 26.30 807063.74 us 150.00 us 68086266.00 us 2438422 MKDIR 49.95 457402.30 us 20.00 us 67894186.00 67894186.00 us8171751 SETATTR* Duration: 353678 seconds Data Read: 21110920120 bytes Data Written: 2338403381483 bytes here is node23 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us 22125898 FORGET 0.00 0.00 us 0.00 us 0.00 us 89286732 RELEASE 0.00 0.00 us 0.00 us 0.00 us 32865496 RELEASEDIR 0.00 35.50 us 23.00 us 48.00 us 2 FGETXATTR 0.00 164.04 us 29.00 us 749181.00 us 39320 FTRUNCATE 0.00 483.71 us 8.00 us 2688755.00 us 39288 LK 0.00 419.61 us 48.00 us 2183971.00 us 274939 LINK 0.00 970.55 us 145.00 us 2471745.00 us 293435 RENAME 0.001346.63 us 35.00 us 4462970.00 us 243238 SETATTR 0.01 285.51 us 25.00 us 2588685.00 us3459436 SETXATTR 0.03 323.11 us 5.00 us 2074581.00 us6977304 READDIR 0.05 12200.60 us 84.00 us 3943421.00 us 287979 RMDIR 0.07 592.75 us 7.00 us 3592073.00 us8129847 STAT 0.076938.50 us 49.00 us 3268036.00 us 705818 UNLINK 0.08 19468.78 us 149.00 us 3664022.00 us 276310 MKNOD 0.09 763.31 us 8.00 us 3396903.00 us8731725 STATFS 0.091715.79 us 4.00 us 5626912.00 us3902746 FLUSH 0.104614.74 us 9.00 us 5835691.00 us1574923 FSTAT 0.101189.55 us 13.00 us 6043163.00 us6129885 OPENDIR 0.10 19729.66 us 131.00 us 4112832.00 us 376286 CREATE 0.13 328.26 us 24.00 us 2410049.00 us 29091424 WRITE 0.202107.64 us 10.00 us 5765196.00 us6675496 GETXATTR 0.285317.38 us 14.00 us 7549301.00 us
Re: [Gluster-users] intresting issue of replication and self-heal
I profiled node22, I found that most latency comes from setxattr, where node23 node22 comes from lookup and locks. any one could help? %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us2437540 FORGET 0.00 0.00 us 0.00 us 0.00 us 252684 RELEASE 0.00 0.00 us 0.00 us 0.00 us2226292 RELEASEDIR 0.00 38.00 us 37.00 us 40.00 us 4 FGETXATTR 0.00 66.16 us 15.00 us 13139.00 us596 GETXATTR 0.00 239.14 us 58.00 us 126477.00 us 1967 LINK 0.00 51.85 us 14.00 us8298.00 us 19045 STAT 0.00 165.50 us 9.00 us 212057.00 us 20544 READDIR 0.001827.92 us 184.00 us 150298.00 us 2084 RENAME 0.00 49.14 us 12.00 us5908.00 us 189019 STATFS 0.00 84.63 us 14.00 us 96016.00 us 163405 READ 0.00 29968.76 us 156.00 us 1073902.00 us 3115 CREATE 0.001340.25 us 6.00 us 7415357.00 us 248141 FLUSH 0.001616.76 us 32.00 us 13865122.00 us 229190 FTRUNCATE 0.011807.58 us 19.00 us 55480776.00 us 249569 OPEN 0.011875.11 us 10.00 us 8842171.00 us 465197 FSTAT 0.05 393296.28 us 52.00 us 56856581.00 us 9057 UNLINK 0.07 32291.01 us 192.00 us 9638107.00 us 156081 RMDIR 0.08 18339.18 us 140.00 us 5313885.00 us 337862 MKNOD 0.092904.39 us 18.00 us 51724741.00 us2226290 OPENDIR 0.154708.15 us 27.00 us 55115760.00 us2334864 SETXATTR 0.188965.91 us 68.00 us 26465968.00 us1513280 FXATTROP 0.213465.29 us 74.00 us 58580783.00 us4506602 XATTROP 0.284801.16 us 44.00 us 49643138.00 us4436847 READDIRP 0.375935.92 us 7.00 us 56449083.00 us4611760 ENTRYLK 1.024226.58 us 33.00 us 63494729.00 us 18092335 WRITE 1.502734.50 us 6.00 us 185109908.00 us 40971541 INODELK 4.75 348602.30 us 5.00 us 2185602946.00 us1019332 FINODELK * 14.98 33957.49 us 14.00 us 59261447.00 us 32998211 LOOKUP 26.30 807063.74 us 150.00 us 68086266.00 us 2438422 MKDIR 49.95 457402.30 us 20.00 us 67894186.00 us8171751 SETATTR* Duration: 353678 seconds Data Read: 21110920120 bytes Data Written: 2338403381483 bytes here is node23 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us 22125898 FORGET 0.00 0.00 us 0.00 us 0.00 us 89286732 RELEASE 0.00 0.00 us 0.00 us 0.00 us 32865496 RELEASEDIR 0.00 35.50 us 23.00 us 48.00 us 2 FGETXATTR 0.00 164.04 us 29.00 us 749181.00 us 39320 FTRUNCATE 0.00 483.71 us 8.00 us 2688755.00 us 39288 LK 0.00 419.61 us 48.00 us 2183971.00 us 274939 LINK 0.00 970.55 us 145.00 us 2471745.00 us 293435 RENAME 0.001346.63 us 35.00 us 4462970.00 us 243238 SETATTR 0.01 285.51 us 25.00 us 2588685.00 us3459436 SETXATTR 0.03 323.11 us 5.00 us 2074581.00 us6977304 READDIR 0.05 12200.60 us 84.00 us 3943421.00 us 287979 RMDIR 0.07 592.75 us 7.00 us 3592073.00 us8129847 STAT 0.076938.50 us 49.00 us 3268036.00 us 705818 UNLINK 0.08 19468.78 us 149.00 us 3664022.00 us 276310 MKNOD 0.09 763.31 us 8.00 us 3396903.00 us8731725 STATFS 0.091715.79 us 4.00 us 5626912.00 us3902746 FLUSH 0.104614.74 us 9.00 us 5835691.00 us1574923 FSTAT 0.101189.55 us 13.00 us 6043163.00 us6129885 OPENDIR 0.10 19729.66 us 131.00 us 4112832.00 us 376286 CREATE 0.13 328.26 us 24.00 us 2410049.00 us 29091424 WRITE 0.202107.64 us 10.00 us 5765196.00 us6675496 GETXATTR 0.285317.38 us 14.00 us 7549301.00 us3798543 OPEN 0.717042.79 us 47.00 us 5848284.00 us7125716 READDIRP 0.80 743.88 us 10.00 us 7979373.00 us 76781383 READ 0.931802.29 us 60.00 us 11040319.00 us 36501360 FXATTROP 1.76 36083.12 us 141.00 us 3548175.00 us3458135 MKDIR 1.835046.35 us 70.00 us 8120221.00 us
[Gluster-users] intresting issue of replication and self-heal
I have a volume (distribute-replica (*3)), today i found an interesting problem node22 node23 and node24 are the replica-7 from client A but the annoying thing is when I create dir or write file from client to replica-7, date;dd if=/dev/zero of=49 bs=1MB count=120 Wed Jan 22 11:51:41 CST 2014 120+0 records in 120+0 records out 12000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s but I could only find node23 node24 have the find --- node23,node24 --- /mnt/xfsd/test-volume/test/49 in clientA, I use find command I use another machine as client B, and mount the test volume (newly mounted) to run* find /mnt/xfsd/test-volume/test/49* from Client A, the three nodes have the file now. --- node22,node23.node24 --- /mnt/xfsd/test-volume/test/49 but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22 still have the file in brick. --- node22 --- /mnt/xfsd/test-volume/test/49 but if i delete the new created files from Client B ) my question is why node22 have no newly created/write dirs/files? I have to use find to trigger the self-heal to fix that? from ClientA's log, I find something like: I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7: no active sinks for performing self-heal on file /test/49 It is harmless for it is information level? I also see something like: [2014-01-19 10:23:48.422757] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-test-volume-replicate-7: Non Blocking entrylks failed for /test/video/2014/01. [2014-01-19 10:23:48.423042] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-test-volume-replicate-7: background entry self-heal failed on /test/video/2014/01 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] intresting issue of replication and self-heal
I have a volume (distribute-replica (*3)), today i found an interesting problem node22 node23 and node24 are the replica-7 from client A but the annoying thing is when I create dir or write file from client to replica-7, date;dd if=/dev/zero of=49 bs=1MB count=120 Wed Jan 22 11:51:41 CST 2014 120+0 records in 120+0 records out 12000 bytes (120 MB) copied, 1.96257 s, 61.1 MB/s but I could only find node23 node24 have the find --- node23,node24 --- /mnt/xfsd/test-volume/test/49 in clientA, I use find command I use another machine as client B, and mount the test volume (newly mounted) to run* find /mnt/xfsd/test-volume/test/49* from Client A, the three nodes have the file now. --- node22,node23.node24 --- /mnt/xfsd/test-volume/test/49 but in Client A, I delete the file /mnt/xfsd/test-volume/test/49, node22 still have the file in brick. --- node22 --- /mnt/xfsd/test-volume/test/49 but if i delete the new created files from Client B ) my question is why node22 have no newly created/write dirs/files? I have to use find to trigger the self-heal to fix that? from ClientA's log, I find something like: I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-test-volume-replicate-7: no active sinks for performing self-heal on file /test/49 It is harmless for it is information level? I also saw something like: [2014-01-19 10:23:48.422757] E [afr-self-heal-entry.c:2376:afr_sh_post_nonblocking_entry_cbk] 0-test-volume-replicate-7: Non Blocking entrylks failed for /test/video/2014/01. [2014-01-19 10:23:48.423042] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-test-volume-replicate-7: background entry self-heal failed on /test/video/2014/01 Is is relative to this issue? ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
