Re: [Gluster-users] Gluster Documentation Update
On Mon, Dec 14, 2015 at 9:47 PM, Joe Julian wrote: > > > On 12/14/2015 07:55 AM, Shaun McCance wrote: > >> the front page of gluster.org still point to the wiki >> pages for planning, not the glusterfs-specs pages >> > > I'm driving at the moment or I'd just do it myself, but that can be > changed at https://github.com/gluster/glusterweb > Shaun, we'll work on the gluster.org front page separately, that's easy to solve. - amye -- Amye Scavarda | a...@redhat.com | Gluster Community Lead ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to diagnose volume rebalance failure?
Hi PuYun, We need to figure out some mechanism to get the huge log files. Until then here is something I can think can be reason that can affect the performance. The rebalance normally starts in medium level [performance wise] which means for you in this case will generate two threads for migration which can hog on those 2 cores. In case you run rebalance again, run it in lazy mode. Here is the command. "gluster v set rebal-throttle lazy". This should spawn just one thread for migration. For logs: Can you grep for errors in the rebalance log file and upload? Thanks, Susant - Original Message - From: "PuYun" To: "gluster-users" Sent: Tuesday, 15 December, 2015 5:51:00 AM Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? Hi, Another weird piece of infomation may be useful. The failed task had actually been running for hours, but the status command output only 3621 sec. == shell == [root@d001 glusterfs]# gluster volume rebalance FastVol status Node Rebalanced-files size scanned failures skipped status run time in secs - --- --- --- --- --- -- localhost 0 0Bytes 952767 0 0 failed 3621.00 volume rebalance: FastVol: success: As you can see, I started rebalance task for only 1 time. cmd_history.log-20151215 == [2015-12-14 12:50:41.443937] : volume start FastVol : SUCCESS [2015-12-14 12:55:01.367519] : volume rebalance FastVol start : SUCCESS [2015-12-14 13:55:22.132199] : volume rebalance FastVol status : SUCCESS [2015-12-14 23:04:01.780885] : volume rebalance FastVol status : SUCCESS [2015-12-14 23:35:56.708077] : volume rebalance FastVol status : SUCCESS = Because the task failed at [ 2015-12-14 21:46:54.179xx], something wrong might happened at 3621 secs before, that is [ 2015-12-14 20:46:33.179xx]. I check logs at that time, found nothing special. == FastVol-rebalance.log [2015-12-14 20:46:33.166748] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/userPoint: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.171009] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjjJ/t2/n1/VSXZlm65KjfhbgoM/flag_finished from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.174851] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_origin.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.181448] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/userPoint from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.184996] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_small.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.191681] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_origin.jpg from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.195396] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_big_square.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 == And, there is no logs around at [ 2015-12-14 20:46:33.179xx ] in mnt-b1-brick.log, mnt-c1-brick.log and etc-glusterfs-glusterd.vol.log. PuYun From: PuYun Date: 2015-12-15 07:30 To: gluster-users Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? Hi, Failed again. I can see disconnections in logs, but no more details. === mnt-b1-brick.log === [2015-12-14 21:46:54.179662] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-FastVol-server: disconnecting connection from d001-1799-2015/12/14-12:54:56:347561-FastVol-client-1-0-0 [2015-12-14 21:46:54.181764] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on / [2015-12-14 21:46:54.181815] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir [2015-12-14 21:46:54.181856] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user [2015-12-14 21:46:54.181918] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/0.jpg [2015-12-14 21:46:5
[Gluster-users] Heavy performance impact to local access (glusterfs 3.6.7)
I have a setup with 2 GlusterFS 3.6.7 servers that are connected with a WAN-connection: root@r1:/daten_gluster# gluster pool list UUID Hostname State 6b70b66c-866f-4222-826b-736a21a9fce1 willy Connected f1ba0eb9-b991-4c99-a177-a4ca7764ff52 localhost Connected PING willy (10.8.0.186) 56(84) bytes of data. 64 bytes from willy (10.8.0.186): icmp_seq=1 ttl=63 time=181 ms 64 bytes from willy (10.8.0.186): icmp_seq=2 ttl=63 time=69.0 ms 64 bytes from willy (10.8.0.186): icmp_seq=3 ttl=63 time=72.1 ms 64 bytes from willy (10.8.0.186): icmp_seq=4 ttl=63 time=71.1 ms 64 bytes from willy (10.8.0.186): icmp_seq=5 ttl=63 time=70.2 ms As you see it's a typical WAN-connect with a latency of about 70ms. And one shared gluster volume: root@r1:/daten_gluster# gluster volume info Volume Name: gv0 Type: Distribute Volume ID: 5baeef5e-4fd4-472f-b313-b0fcd1baa17a Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: r1:/storage/gluster Brick2: willy:/storage/gluster Options Reconfigured: nfs.export-volumes: off cluster.readdir-optimize: on performance.readdir-ahead: on diagnostics.brick-log-level: WARNING diagnostics.client-log-level: WARNING performance.write-behind-window-size: 64MB performance.cache-size: 256MB performance.client-io-threads: on performance.cache-refresh-timeout: 10 nfs.addr-namelookup: off cluster.min-free-disk: 1 cluster.data-self-heal-algorithm: full performance.io-thread-count: 64 nfs.disable: true performance.flush-behind: on As you see I tried all performance-options I found that might be useful. The problem is, when working in the gluster-mounted directory (using -t glusterfs, I tried NFS too, but there was not that great performance win), _EVERYTHING_ is DEAD SLOW: root@r1:/daten_gluster# time find test test test/4 test/4/file05 test/4/file04 test/4/file02 test/4/file03 test/4/file01 test/2 test/2/file05 test/2/file04 test/2/file02 test/2/file03 test/2/file01 test/file05 test/3 test/3/file05 test/3/file04 test/3/file02 test/3/file03 test/3/file01 test/file04 test/file02 test/file03 test/1 test/1/file05 test/1/file04 test/1/file02 test/1/file03 test/1/file01 test/file01 real0m4.734s user0m0.000s sys 0m0.000s When I disconnect the other node (willy): root@r1:/daten_gluster# gluster volume remove-brick gv0 willy:/storage/gluster force Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y volume remove-brick commit force: success root@r1:/daten_gluster# time find test test test/4 test/4/file05 test/4/file04 test/4/file02 test/4/file03 test/4/file01 test/2 test/2/file05 test/2/file04 test/2/file02 test/2/file03 test/2/file01 test/file05 test/3 test/3/file05 test/3/file04 test/3/file02 test/3/file03 test/3/file01 test/file04 test/file02 test/file03 test/1 test/1/file05 test/1/file04 test/1/file02 test/1/file03 test/1/file01 test/file01 real0m0.017s user0m0.000s sys 0m0.000s 5 secs compared to 0,02 secs WHY? I'm just running local reads... (not even writes that might be distributed) When I add willy again it again slows down to death: root@r1:/daten_gluster# gluster volume add-brick gv0 willy:/storage/gluster force volume add-brick: success root@r1:/daten_gluster# time find test test test/4 test/4/file05 test/4/file04 test/4/file02 test/4/file03 test/4/file01 test/2 test/2/file05 test/2/file04 test/2/file02 test/2/file03 test/2/file01 test/file05 test/3 test/3/file05 test/3/file04 test/3/file02 test/3/file03 test/3/file01 test/file04 test/file02 test/file03 test/1 test/1/file05 test/1/file04 test/1/file02 test/1/file03 test/1/file01 test/file01 real0m5.226s user0m0.000s sys 0m0.000s These were only 30 files. I wanted to share 220.000 files with glusterfs... Where is my configuration-mistake? Can you please help me or give me a hint? I cannot belive that GlusterFS is that problematic on WAN-connects...? Thank you very much! Joerg ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] libgfapi access
On 12/11/2015 08:58 PM, Ankireddypalle Reddy wrote: Pranith, Thanks for checking this. Though the time taken to run was 18 seconds if you look at the time consumed in user land as well as kernel land for executing the command then it is evident that fuse took almost half the time as libgfapi. Also from the collected profiles it is evident that the average latency for the write command is less for fuse than for libgfapi. Are there any recommendations for I/O through libgfapi for disperse volumes. Is there any way to avoid the extra memcpy's that are being made when performing I/O through libgfapi. hi Ankireddy, Oh this is not a problem. If we use fuse, the system call 'write' from ./GlusterFuseTest will go through fuse-kernel, fuse kernel sends the write operation to glusterfs mount process which is a user process. Time taken to complete that call from then on is computed against the glusterfs mount process until it responds to the fuse-kernel, not against the ./GlusterFuseTest process. If we use gfapi, there is no system call over head, instead ./GlusterFuseTest process directly makes calls with the bricks through gfapi library. So all the time that the process spends communicating with the bricks and getting the response is counted against ./GlusterFuseTest. That is the reason you see more 'user' time. So again, There are quite a few workloads where gfapi has proven to give better response times than fuse mounts because we avoid the context switch costs of ./GlusterFuseTest -> fuse-kernel -> glusterfs-mount -> fuse-kernel (for response)-> ./GlusterFuseTest (for response to 'write') Hope that helps. Sorry for the delay in response, was in too many meetings yesterday. Pranith Thanks and Regards, Ram -Original Message- From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com] Sent: Thursday, December 10, 2015 10:57 PM To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org Subject: Re: [Gluster-users] libgfapi access On 12/10/2015 07:15 PM, Ankireddypalle Reddy wrote: Hi, Please let me know in case you need any more details. Even for only write operations fuse seems to outperform libgfapi. Is it because of disperse volumes?. Also I noticed a lot of data loss in case I use libgfapi asyn I/O for disperse volumes. Fuse and gfapi seem to take same amount of time to complete the run, i.e. 18 seconds. Could you let me know what you mean by fuse outperforming gfapi? Pranith Thanks and Regards, Ram -Original Message- From: Ankireddypalle Reddy Sent: Wednesday, December 09, 2015 5:01 PM To: 'Pranith Kumar Karampuri'; Vijay Bellur; gluster-users@gluster.org Subject: RE: [Gluster-users] libgfapi access Hi, I upgraded my setup to gluster 3.7.3. I tested writes by performing writes through fuse and through libgfapi. Attached are the profiles generated from fuse and libgfapi. The test programs essentially writes 1 blocks each of 128K. [root@santest2 Base]# time ./GlusterFuseTest /ws/glus 131072 1 Mount path: /ws/glus Block size: 131072 Num of blocks: 1 Will perform write test on mount path : /ws/glus Succesfully created file /ws/glus/1449697583.glfs Successfully filled file /ws/glus/1449697583.glfs Write test succeeded Write test succeeded. real0m18.722s user0m3.913s sys 0m1.126s [root@santest2 Base]# time ./GlusterLibGFApiTest dispersevol santest2 24007 131072 1 Host name: santest2 Volume: dispersevol Port: 24007 Block size: 131072 Num of blocks: 1 Will perform write test on volume: dispersevol Successfully filled file 1449697651.glfs Write test succeeded Write test succeeded. real0m18.630s user0m8.804s sys 0m1.870s Thanks and Regards, Ram -Original Message- From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com] Sent: Wednesday, December 09, 2015 1:39 AM To: Ankireddypalle Reddy; Vijay Bellur; gluster-users@gluster.org Subject: Re: [Gluster-users] libgfapi access On 12/08/2015 08:28 PM, Ankireddypalle Reddy wrote: Vijay, We are trying to write data backed up by Commvault simpana to glusterfs volume. The data being written is around 30 GB. Two kinds of write requests happen. 1) 1MB requests 2) Small write requests of size 128 bytes. In case of libgfapi access these are cached and a single 128KB write request is made where as in case of FUSE the 128 byte write request is handled to FUSE directly. glusterfs 3.6.5 built on Aug 24 2015 10:02:43 Volume Name: dispersevol Type: Disperse Volume ID: c5d6ccf8-6fec-4912-ab2e-6a7701e4c4c0 Status: Started Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: ssdtest:/mnt/ssdfs1/brick3 Brick2: sanserver2:/data/brick3 Brick3: santest2:/home/brick3 Options Reconfigured: performance.cache-size: 512MB performance.writ
[Gluster-users] No longer possible to "recreate" a GlusterFS "Distributed-Replicate" volume from its bricks?
It seems that it is no longer possible to stop, and delete a GlusterFS volume, and then clean, and recreate it from the component bricks, when the volume is "Distributed-Replicate". The same steps used with a simple "Replicate" or "Distribute" volume are successful with the correct data. For the failing "Distributed-Replicate" case, the "rebalance" always fails after the recreate. The "heal" command is successful, showing no "info split-brain" files indicated, but about half of the files are missing, and there are many "(Possible split-brain)" warnings in the logfile. The gluster "Distributed-Replicate" volume "recreate" procedure works fine in GlusterFS versions 3.2.7, and 3.4.2, but not in glusterfs 3.6.5, 3.6.7, or 3.7.6. Perhaps the "recreate" procedure has changed, or I'm doing something wrong that now matters in the newer GlusterFS versions. Details below. Any ideas how to make it work again? Thanks. ~ Jeff Byers ~ # glusterd -V glusterfs 3.7.6 built on Dec 14 2015 07:05:12 # Failing "Distributed-Replicate" recreate case. # mountpoint /exports/test-dir/ /exports/test-dir/ is a mountpoint # mount |grep test-dir /dev/sdu on /exports/test-dir type xfs (rw,noatime,nodiratime,barrier,nouuid,inode64,logbufs=8,logbsize=256k) # mkdir /exports/test-dir/test-brick-1a # mkdir /exports/test-dir/test-brick-1b # mkdir /exports/test-dir/test-brick-2a # mkdir /exports/test-dir/test-brick-2b # gluster volume create test-replica-dist replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-1a 10.10.60.169:/exports/test-dir/test-brick-2a 10.10.60.169:/exports/test-dir/test-brick-1b 10.10.60.169:/exports/test-dir/test-brick-2b force volume create: test-replica-dist: success: please start the volume to access data # gluster volume start test-replica-dist volume start: test-replica-dist: success # gluster volume info test-replica-dist Volume Name: test-replica-dist Type: Distributed-Replicate Volume ID: c8de4e65-2304-4801-a244-6511f39fc0c9 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.10.60.169:/exports/test-dir/test-brick-1a Brick2: 10.10.60.169:/exports/test-dir/test-brick-2a Brick3: 10.10.60.169:/exports/test-dir/test-brick-1b Brick4: 10.10.60.169:/exports/test-dir/test-brick-2b Options Reconfigured: snap-activate-on-create: enable # mkdir /mnt/test-replica-dist # mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica-dist /mnt/test-replica-dist/ # cp -rf /lib64/ /mnt/test-replica-dist/ # diff -r /lib64/ /mnt/test-replica-dist/lib64/ # umount /mnt/test-replica-dist # gluster volume stop test-replica-dist Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: test-replica-dist: success # gluster volume delete test-replica-dist Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y volume delete: test-replica-dist: success # gluster_clear_xattrs.sh /exports/test-dir/test-brick-1a removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1a xattr clean-up in progress: /exports/test-dir/test-brick-1a /exports/test-dir/test-brick-1a ready to be used as a glusterfs brick # gluster_clear_xattrs.sh /exports/test-dir/test-brick-1b removing all .glusterfs directories in progress: /exports/test-dir/test-brick-1b xattr clean-up in progress: /exports/test-dir/test-brick-1b /exports/test-dir/test-brick-1b ready to be used as a glusterfs brick # gluster_clear_xattrs.sh /exports/test-dir/test-brick-2a removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2a xattr clean-up in progress: /exports/test-dir/test-brick-2a /exports/test-dir/test-brick-2a ready to be used as a glusterfs brick # gluster_clear_xattrs.sh /exports/test-dir/test-brick-2b removing all .glusterfs directories in progress: /exports/test-dir/test-brick-2b xattr clean-up in progress: /exports/test-dir/test-brick-2b /exports/test-dir/test-brick-2b ready to be used as a glusterfs brick # gluster volume create test-replica-dist replica 2 transport tcp 10.10.60.169:/exports/test-dir/test-brick-1a 10.10.60.169:/exports/test-dir/test-brick-2a 10.10.60.169:/exports/test-dir/test-brick-1b 10.10.60.169:/exports/test-dir/test-brick-2b force volume create: test-replica-dist: success: please start the volume to access data # gluster volume start test-replica-dist volume start: test-replica-dist: success # mount -t glusterfs -o acl,log-level=WARNING 127.0.0.1:/test-replica-dist /mnt/test-replica-dist/ # diff -r /lib64/ /mnt/test-replica-dist/lib64/ Only in /lib64/device-mapper: libdevmapper-event-lvm2thin.so Only in /lib64/multipath: libcheckcciss_tur.so Only in /lib64/multipath: libcheckemc_clariion.so Only in /lib64/multipath: libcheckhp_sw.so Only in /lib64/multipath: libprioconst.so Only in /lib64/multipath: libpriordac.so Only in /lib64/multipath: libprioweighted.so Only in /lib64/rtkai
Re: [Gluster-users] How to diagnose volume rebalance failure?
Hi, Another weird piece of infomation may be useful. The failed task had actually been running for hours, but the status command output only 3621 sec. == shell == [root@d001 glusterfs]# gluster volume rebalance FastVol status Node Rebalanced-files size scanned failures skipped status run time in secs - --- --- --- --- --- -- localhost00Bytes 952767 0 0 failed3621.00 volume rebalance: FastVol: success: As you can see, I started rebalance task for only 1 time. cmd_history.log-20151215 == [2015-12-14 12:50:41.443937] : volume start FastVol : SUCCESS [2015-12-14 12:55:01.367519] : volume rebalance FastVol start : SUCCESS [2015-12-14 13:55:22.132199] : volume rebalance FastVol status : SUCCESS [2015-12-14 23:04:01.780885] : volume rebalance FastVol status : SUCCESS [2015-12-14 23:35:56.708077] : volume rebalance FastVol status : SUCCESS = Because the task failed at [2015-12-14 21:46:54.179xx], something wrong might happened at 3621 secs before, that is [2015-12-14 20:46:33.179xx]. I check logs at that time, found nothing special. == FastVol-rebalance.log [2015-12-14 20:46:33.166748] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/userPoint: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.171009] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjjJ/t2/n1/VSXZlm65KjfhbgoM/flag_finished from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.174851] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_origin.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.181448] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/userPoint from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.184996] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_small.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.191681] I [MSGID: 109022] [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration of /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_origin.jpg from subvolume FastVol-client-0 to FastVol-client-1 [2015-12-14 20:46:33.195396] I [dht-rebalance.c:1010:dht_migrate_file] 0-FastVol-dht: /for_ybest_fsdir/user/Weixin.oClDcjjJ/rH/wV/mNv6sX94lypFWdvM/portrait_big_square.jpg: attempting to move from FastVol-client-0 to FastVol-client-1 == And, there is no logs around at [2015-12-14 20:46:33.179xx] in mnt-b1-brick.log, mnt-c1-brick.log and etc-glusterfs-glusterd.vol.log. PuYun From: PuYun Date: 2015-12-15 07:30 To: gluster-users Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? Hi, Failed again. I can see disconnections in logs, but no more details. === mnt-b1-brick.log === [2015-12-14 21:46:54.179662] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-FastVol-server: disconnecting connection from d001-1799-2015/12/14-12:54:56:347561-FastVol-client-1-0-0 [2015-12-14 21:46:54.181764] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on / [2015-12-14 21:46:54.181815] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir [2015-12-14 21:46:54.181856] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user [2015-12-14 21:46:54.181918] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/0.jpg [2015-12-14 21:46:54.181961] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user/ji/ay/an [2015-12-14 21:46:54.182003] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/icon_loading_white22c04a.gif [2015-12-14 21:46:54.182036] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /for_ybest_fsdir/user/ji [2015-12-14 21:46:54.182076] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cl
Re: [Gluster-users] How to diagnose volume rebalance failure?
Hi,
Failed again. I can see disconnections in logs, but no more details.
=== mnt-b1-brick.log ===
[2015-12-14 21:46:54.179662] I [MSGID: 115036] [server.c:552:server_rpc_notify]
0-FastVol-server: disconnecting connection from
d001-1799-2015/12/14-12:54:56:347561-FastVol-client-1-0-0
[2015-12-14 21:46:54.181764] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /
[2015-12-14 21:46:54.181815] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir
[2015-12-14 21:46:54.181856] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user
[2015-12-14 21:46:54.181918] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/0.jpg
[2015-12-14 21:46:54.181961] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/an
[2015-12-14 21:46:54.182003] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/icon_loading_white22c04a.gif
[2015-12-14 21:46:54.182036] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji
[2015-12-14 21:46:54.182076] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay
[2015-12-14 21:46:54.182110] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/an/ling00
[2015-12-14 21:46:54.182203] I [MSGID: 101055] [client_t.c:419:gf_client_unref]
0-FastVol-server: Shutting down connection
d001-1799-2015/12/14-12:54:56:347561-FastVol-client-1-0-0
==
== mnt-c1-brick.log -
[2015-12-14 21:46:54.179597] I [MSGID: 115036] [server.c:552:server_rpc_notify]
0-FastVol-server: disconnecting connection from
d001-1799-2015/12/14-12:54:56:347561-FastVol-client-0-0-0
[2015-12-14 21:46:54.180428] W [inodelk.c:404:pl_inodelk_log_cleanup]
0-FastVol-server: releasing lock on 5e300cdb-7298-44c0-90eb-5b50018daed6 held
by {client=0x7effc810cce0, pid=-3 lk-owner=fdff}
[2015-12-14 21:46:54.180454] W [inodelk.c:404:pl_inodelk_log_cleanup]
0-FastVol-server: releasing lock on 3c9a1cd5-84c8-4967-98d5-e75a402b1f74 held
by {client=0x7effc810cce0, pid=-3 lk-owner=fdff}
[2015-12-14 21:46:54.180483] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on /
[2015-12-14 21:46:54.180525] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir
[2015-12-14 21:46:54.180570] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user
[2015-12-14 21:46:54.180604] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/0.jpg
[2015-12-14 21:46:54.180634] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji
[2015-12-14 21:46:54.180678] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay
[2015-12-14 21:46:54.180725] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/an/ling00
[2015-12-14 21:46:54.180779] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/up/a19640529/linkwrap/129836/icon_loading_white22c04a.gif
[2015-12-14 21:46:54.180820] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup] 0-FastVol-server: fd cleanup on
/for_ybest_fsdir/user/ji/ay/an
[2015-12-14 21:46:54.180859] I [MSGID: 101055] [client_t.c:419:gf_client_unref]
0-FastVol-server: Shutting down connection
d001-1799-2015/12/14-12:54:56:347561-FastVol-client-0-0-0
==
== etc-glusterfs-glusterd.vol.log ==
[2015-12-14 21:46:54.179819] W [socket.c:588:__socket_rwv] 0-management: readv
on /var/run/gluster/gluster-rebalance-dbee250a-e3fe-4448-b905-b76c5ba80b25.sock
failed (No data available)
[2015-12-14 21:46:54.209586] I [MSGID: 106007]
[glusterd-rebalance.c:162:__glusterd_defrag_notify] 0-management: Rebalance
process for volume FastVol has disconnected.
[2015-12-14 21:46:54.209627] I [MSGID: 101053]
[mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=1 total=1
[2015-12-14 21:46:54.209640] I [MSGID: 101053]
[mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=1 total=1
=
== FastVol-rebalance.log
...
[2015-12-14 21:46:53.423719] I [MSGID: 109022]
[dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht:
Re: [Gluster-users] Issue with storage access when brick goes down
I thought it was to do with the expense of tearing down and setting up the connections, so the timeout is there to avoid an expensive operation if it is not really necessary. On 7 December 2015 at 22:15, Bipin Kunal wrote: > I assume that this is because the host which went down is even the host > which is used for mounting client. > > Suppose there are 2 host. Host1 and Host2. And client is mounted as > mount -t glusterfs host1:brick1 /mnt > > In this case if host1 goes down, client will wait till network ping > timeout before it starts accessing volume using other host(host2). > > So I think this is expected behaviour. > > Thanks, > Bipin Kunal > On Dec 7, 2015 10:57 PM, "L, Sridhar (Nokia - IN/Bangalore)" < > sridha...@nokia.com> wrote: > >> Hello, >> >> I am running gluster storage in distributed replicated mode. When one of >> the brick (host) goes offline, operations on the file system by the client >> will hang for a while and resumes after sometime. I searched and found that >> operations hang for the period set in network.ping-timeout. >> Can anyone explain me why the client operations will hang even though the >> other brick is available with all the data? >> >> >> Regards, >> Sridhar L >> >> >> >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster Documentation Update
On 12/14/2015 07:55 AM, Shaun McCance wrote: the front page of gluster.org still point to the wiki pages for planning, not the glusterfs-specs pages I'm driving at the moment or I'd just do it myself, but that can be changed at https://github.com/gluster/glusterweb ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster Documentation Update
On Thu, 2015-12-10 at 08:34 +0530, Humble Devassy Chirammal wrote: > Above is a deadlink because "Features" directory itself is moved and > its now part of https://github.com/gluster/glusterfs-specs/blob/maste > r/done/Features/quota-object-count.md What's the long-term plan for these specs? It's useful to be able to point people to a URL for a spec where they can read it rendered. Do we plan on publishing them somehow, or is the GitHub preview for Markdown enough? Right now, the specs are organized in done and in_progress directories. I understand this helps organize, but it makes it very difficult to have links from other places that don't constantly break. Before I realized you'd already migrated these to the glusterfs-specs repository, I was working on moving them to the GitHub wiki on the glusterfs repository. See this for example: https://github.com/shaunix/glusterfs/wiki/Fault-Injection I'd really like to have redirects from the old MediaWiki URLs to the new specs. Also, the front page of gluster.org still point to the wiki pages for planning, not the glusterfs-specs pages. We need stable URLs to point people to. -- Shaun ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] All issues resolved after disabling RDMA
I have been experiencing several issues with glusterfs for several months These started more or less after upgrading to release 3.7 from 3.5. I skipped 3.6 series. Almost at the same time I had introduced and Infiniband point to point network between my two gluster bricks. The symptoms were failures to start the volume even when both nodes were up and running, failed synchronizations, unexplicable split-brain even for those files that I had certainty were only accessed by a single client only. I was about to give up glusterfs altogether. As a last resort first I tried again disabling RDMA (over infiniband) and I rebuilt the bricks from scratch using only TCP from the start (I had tried before to disable RDMA, but without starting from scratch, so I must have experienced what were latent issues). I cannot tell whether the RDMA defects are caused by gluster, the hardware or the operating system, however now using TCP only over infiniband I have had a stable cluster with an active node and a second node which I usually leave turned off and that synchronizes perfectly every time I turn it back on. I hope this helps. Mauro ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to diagnose volume rebalance failure?
Hi, Thank you for your reply. I don't know how to send you the huge sized rebalance log file which is about 2GB. However, I might have found out the reason why the task failed. My gluster server has only 2 cpu cores and carries 2 ssd bricks. When the rebalance task began, top 3 processes are 70%~80%, 30%~40 and 30%~40 cpu usage. Others are less than 1%. But after a while, 2 CPU cores are used up totally and I even can't login until the rebalance task failed. It seems 2 bricks require 4 CPU cores at least. Now I upgrade the virtual server with 8 CPU cores and start rebalance task again. Everything goes well for now. I will report again when the current task completed or failed. PuYun From: Nithya Balachandran Date: 2015-12-14 18:57 To: PuYun CC: gluster-users Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? Hi, Can you send us the rebalance log? Regards, Nithya - Original Message - > From: "PuYun" > To: "gluster-users" > Sent: Monday, December 14, 2015 11:33:40 AM > Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? > > Here is the tail of the failed rebalance log, any clue? > > [2015-12-13 21:30:31.527493] I [dht-rebalance.c:2340:gf_defrag_process_dir] > 0-FastVol-dht: Migration operation on dir > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/5F/1MsH5--BcoGRAJPI took 20.95 secs > [2015-12-13 21:30:31.528704] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:30:31.543901] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:31:37.210496] I [MSGID: 109081] > [dht-common.c:3780:dht_setxattr] 0-FastVol-dht: fixing the layout of > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q > [2015-12-13 21:31:37.722825] I [MSGID: 109045] > [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 0 > (FastVol-client-0): 1032124 chunks > [2015-12-13 21:31:37.722837] I [MSGID: 109045] > [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 1 > (FastVol-client-1): 1032124 chunks > [2015-12-13 21:33:03.955539] I [MSGID: 109064] > [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: > FastVol-client-0; inode layout - 0 - 2146817919 - 1; disk layout - > 2146817920 - 4294967295 - 1 > [2015-12-13 21:33:04.069859] I [MSGID: 109018] > [dht-common.c:806:dht_revalidate_cbk] 0-FastVol-dht: Mismatching layouts for > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q, gfid = > f38c4ed2-a26a-4d83-adfd-6b0331831738 > [2015-12-13 21:33:04.118800] I [MSGID: 109064] > [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: > FastVol-client-1; inode layout - 2146817920 - 4294967295 - 1; disk layout - > 0 - 2146817919 - 1 > [2015-12-13 21:33:19.979507] I [MSGID: 109022] > [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration > of > /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished > from subvolume FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:19.979459] I [MSGID: 109022] > [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration > of /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint > from subvolume FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:25.543941] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_origin.jpg: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:25.962547] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_small.jpg: > attempting to move from FastVol-client-0 to FastVol-client-1 > > > Cloudor > > > > From: Sakshi Bansal > Date: 2015-12-12 13:02 > To: 蒲云 > CC: gluster-users > Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? > In the rebalance log file you can check the file/directory for which the > rebalance has failed. It can mention what was the fop for whihc the failure > happened. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Strange file corruption - it happened again
Hi, it happened again: today I've upgraded some packages on node #3. Since the Kernel had a minor update, I was asked to reboot the server, and did so. At that time only one (non-critical) VM was running on that node. I've checked twice and Gluster was *not* healing when I've rebooted. After rebooting, and while *automatic* healing was in progress, one VM started to get HDD corruption again, up to the point that it wasn't able to boot anymore(!). That poor VM was one of the only two VMs that were still using NFS for accessing the Gluster storage - if that matters. The second VM survived the healing, even if it has rather large disks (~380 GB) and is rather busy. All other ~13 VMs had been moved to native glusterfs mount days before and had no problem with the reboot. The Gluster access type may be related or not - I don't know... All Gluster packages are at version "3.5.2-2+deb8u1" on all three servers - so Gluster has *not* been upgraded this time. Kernel on node #3: Linux metal3 4.2.6-1-pve #1 SMP Wed Dec 9 10:49:55 CET 2015 x86_64 GNU/Linux Kenrle node #1: Linux metal1 4.2.3-2-pve #1 SMP Sun Nov 15 16:08:19 CET 2015 x86_64 GNU/Linux Any idea?? Udo Am 10.12.2015 um 16:12 schrieb Udo Giacomozzi: Am 09.12.2015 um 22:33 schrieb Lindsay Mathieson: On 10/12/2015 3:15 AM, Udo Giacomozzi wrote: This were the commands executed on node #2 during step 6: gluster volume add-brick "systems" replica 3 metal1:/data/gluster/systems gluster volume heal "systems" full # to trigger sync Then I waited for replication to finish before doing anything else (about 1 hour or maybe more), checking _gluster volume heal "systems" info_ Did you execute the heal command from host #2? Might be related to a possible issue I encountered during testing adding bricks recently, still in the process of recreating and testing the issue. I'm afraid I can't tell anymore. Could be, I'm not sure, sorry... Udo ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to diagnose volume rebalance failure?
Hi, Can you send us the rebalance log? Regards, Nithya - Original Message - > From: "PuYun" > To: "gluster-users" > Sent: Monday, December 14, 2015 11:33:40 AM > Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? > > Here is the tail of the failed rebalance log, any clue? > > [2015-12-13 21:30:31.527493] I [dht-rebalance.c:2340:gf_defrag_process_dir] > 0-FastVol-dht: Migration operation on dir > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/5F/1MsH5--BcoGRAJPI took 20.95 secs > [2015-12-13 21:30:31.528704] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:30:31.543901] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:31:37.210496] I [MSGID: 109081] > [dht-common.c:3780:dht_setxattr] 0-FastVol-dht: fixing the layout of > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q > [2015-12-13 21:31:37.722825] I [MSGID: 109045] > [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 0 > (FastVol-client-0): 1032124 chunks > [2015-12-13 21:31:37.722837] I [MSGID: 109045] > [dht-selfheal.c:1508:dht_fix_layout_of_directory] 0-FastVol-dht: subvolume 1 > (FastVol-client-1): 1032124 chunks > [2015-12-13 21:33:03.955539] I [MSGID: 109064] > [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: > FastVol-client-0; inode layout - 0 - 2146817919 - 1; disk layout - > 2146817920 - 4294967295 - 1 > [2015-12-13 21:33:04.069859] I [MSGID: 109018] > [dht-common.c:806:dht_revalidate_cbk] 0-FastVol-dht: Mismatching layouts for > /for_ybest_fsdir/user/Weixin.oClDcjhe/Ny/7Q, gfid = > f38c4ed2-a26a-4d83-adfd-6b0331831738 > [2015-12-13 21:33:04.118800] I [MSGID: 109064] > [dht-layout.c:808:dht_layout_dir_mismatch] 0-FastVol-dht: subvol: > FastVol-client-1; inode layout - 2146817920 - 4294967295 - 1; disk layout - > 0 - 2146817919 - 1 > [2015-12-13 21:33:19.979507] I [MSGID: 109022] > [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration > of > /for_ybest_fsdir/user/Weixin.oClDcjhe/Kn/hM/oHcPMp4hKq5Tq2ZQ/flag_finished > from subvolume FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:19.979459] I [MSGID: 109022] > [dht-rebalance.c:1290:dht_migrate_file] 0-FastVol-dht: completed migration > of /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/userPoint > from subvolume FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:25.543941] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_origin.jpg: > attempting to move from FastVol-client-0 to FastVol-client-1 > [2015-12-13 21:33:25.962547] I [dht-rebalance.c:1010:dht_migrate_file] > 0-FastVol-dht: > /for_ybest_fsdir/user/Weixin.oClDcjhe/PU/ps/qUa-n38i8QBgeMdI/portrait_small.jpg: > attempting to move from FastVol-client-0 to FastVol-client-1 > > > Cloudor > > > > From: Sakshi Bansal > Date: 2015-12-12 13:02 > To: 蒲云 > CC: gluster-users > Subject: Re: [Gluster-users] How to diagnose volume rebalance failure? > In the rebalance log file you can check the file/directory for which the > rebalance has failed. It can mention what was the fop for whihc the failure > happened. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
