Just before updrading to 3.3final, we had an rsync collision on our
gluster filesystem which left us with undeletable dirs.
The transport is IPoIB over 4 bricks as shown below.
$ gluster volume info
Volume Name: gli
Type: Distribute
Volume ID: 76cc5e88-0ac4-42ac-a4a3-31bf2ba611d4
Status: Started
Number of Bricks: 4
Transport-type: tcp,rdma
Bricks:
Brick1: pbs1ib:/bducgl
Brick2: pbs2ib:/bducgl
Brick3: pbs3ib:/bducgl
Brick4: pbs4ib:/bducgl
Options Reconfigured:
performance.io-cache: on
performance.quick-read: on
performance.io-thread-count: 64
The last 100 line of the gluster log is here: <http://pastie.org/4036243>
The problem started when I was running an rsync with --delete while
moving a large /home to the gluster dir to prune the target of older
cruft in a /home move.. A user directed a job to start reading from a
directory that was simultaneously being pruned and all hell broke loose.
Now we're left with a number of dirs and files that can't be deleted.
'du' says there are files in an apparently empty dir:
root@bduc-login:/gl/tvanerp/adni
561 $ du -sh *
40K dicom
altho 'ls' can't see them from the client;
root@bduc-login:/gl/tvanerp/adni
562 $ ls -lat dicom
total 40
drwxrwxr-x 3 tvanerp tvanerp 94214 Jun 5 23:40 ./
drwxrwxr-x 3 tvanerp tvanerp 72 Apr 23 13:11 ../
From the bricks you can see that there are still files there:
Tue Jun 05 23:18:02 [3.06 2.42 1.65][457.48/606] root@bduc-login:~
556 $ ssh pbs2 'ls -lat /bducgl/tvanerp/adni/dicom'
total 24
drwxrwxr-x 3 7335 7335 28672 2012-06-05 22:59 .
drwxr-xr-x 2 7335 7335 8192 2012-06-05 14:50 128_S_0947_20080422_MPRAGE
drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 ..
Tue Jun 05 23:18:26 [2.82 2.42 1.67][457.62/606] root@bduc-login:~
557 $ ssh pbs3 'ls -lat /bducgl/tvanerp/adni/dicom'
total 28
drwxrwxr-x 3 7335 7335 28672 2012-06-05 23:00 .
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:52 128_S_0947_20080422_MPRAGE
drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 ..
Tue Jun 05 23:18:51 [2.73 2.43 1.69][457.7/606] root@bduc-login:~
558 $ ssh pbs4 'ls -lat /bducgl/tvanerp/adni/dicom'
total 32
drwxrwxr-x 3 7335 7335 36864 2012-06-05 22:59 .
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:50 128_S_0947_20080422_MPRAGE
---------T 2 root root 0 2012-05-20 18:35 adni_pib_subjects.txt
drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 ..
but the client is not able to see/delete them, even as root.
Suggestions?
harry
--
Harry Mangalam,Research Computing, OIT,
Rm 225 MSTB, UC Irvine [mailcode 2225]
Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
Lat/Long: (33.642025,-117.844414) [paste into Google Maps]
--
_______________________________________________
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
