Re: [Gluster-users] "Structure needs cleaning" error
Hi Peng, Which version of glusterfs is in use here ? also, which mount options are used to fuse mount this volume? because, I see a similar discussion here , http://gluster.org/pipermail/gluster-users/2013-December/038299.html --Humble On Sat, May 24, 2014 at 11:22 PM, Peng Yu wrote: > Hi, > > I see "Structure needs cleaning" error, when I touch a file on > glusterfs. Does anybody know what causes the problem and how to fix > it? Thanks. > > -- > Regards, > Peng > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] "Structure needs cleaning" error
Hi, I see "Structure needs cleaning" error, when I touch a file on glusterfs. Does anybody know what causes the problem and how to fix it? Thanks. -- Regards, Peng ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
On Mon, Feb 27, 2012 at 6:59 AM, Jeff Darcy wrote: > On Sun, 26 Feb 2012 13:10:06 -0500 > Joe Landman wrote: > >> On 02/26/2012 12:54 PM, Patrick Haley wrote: >> > If I go to that directory and simply do an ls *.png I get >> > >> > ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning >> >> This is usually what happens when you have an underlying xfs file >> system as your backing store, and have had a failure which has shut >> the xfs file system down. > > Actually this error (EUCLEAN) is used in a couple of places in the > GlusterFS code. Specifically it's used as an initial status when AFR > is setting up a structure to track locking state as part of our > replication protocol, and if nothing comes along to change that status > it could "leak" to the caller. I saw this when I was hacking on > replication code, so in that case it was my own fault, but it's not > inconceivable that there's another path which could trigger it. > > Still, though, the XFS explanation seems more likely. I'd pursue that > first. If it was an XFS caused error, there must be a log with that very strerror() "Structure needs cleaning" in the brick logs. Avati ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
Hi, Does anyone have a ballpark guess as to how long an xfs_repair should take for a 15TB area? This will help us schedule the down-time, so we're only looking for rough, order-of-magnitude estimates. Thanks. Hi, It does appear to be xfs /dev/mapper/the_raid-lv_data on /data type xfs (rw) Looking at the man page for xfs_repair indicates that we'll have to unmount the disk first. That will have to wait until we can schedule the down-time. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-222B http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 Quoting Joe Landman : On 02/26/2012 01:35 PM, Patrick Haley wrote: Hi, I went on the particular server on which the file in question resides. I check the RAID and all the disks show up as OK. I looked in dmesg and did not see anything on xfs. To confirm this absence, I did "grep -in xfs dmesg" and that also came up empty. "grep -in fs dmesg" (no "x") returns type mount and see what the file system is mounted as. The structure needs cleaning in xfs means an xfs_repair is needed. Not sure if this is used by other file systems. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-222B http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
On Sun, 26 Feb 2012 13:10:06 -0500 Joe Landman wrote: > On 02/26/2012 12:54 PM, Patrick Haley wrote: > > If I go to that directory and simply do an ls *.png I get > > > > ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning > > This is usually what happens when you have an underlying xfs file > system as your backing store, and have had a failure which has shut > the xfs file system down. Actually this error (EUCLEAN) is used in a couple of places in the GlusterFS code. Specifically it's used as an initial status when AFR is setting up a structure to track locking state as part of our replication protocol, and if nothing comes along to change that status it could "leak" to the caller. I saw this when I was hacking on replication code, so in that case it was my own fault, but it's not inconceivable that there's another path which could trigger it. Still, though, the XFS explanation seems more likely. I'd pursue that first. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
Hi, It does appear to be xfs /dev/mapper/the_raid-lv_data on /data type xfs (rw) Looking at the man page for xfs_repair indicates that we'll have to unmount the disk first. That will have to wait until we can schedule the down-time. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-222B http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 Quoting Joe Landman : On 02/26/2012 01:35 PM, Patrick Haley wrote: Hi, I went on the particular server on which the file in question resides. I check the RAID and all the disks show up as OK. I looked in dmesg and did not see anything on xfs. To confirm this absence, I did "grep -in xfs dmesg" and that also came up empty. "grep -in fs dmesg" (no "x") returns type mount and see what the file system is mounted as. The structure needs cleaning in xfs means an xfs_repair is needed. Not sure if this is used by other file systems. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
On 02/26/2012 01:35 PM, Patrick Haley wrote: Hi, I went on the particular server on which the file in question resides. I check the RAID and all the disks show up as OK. I looked in dmesg and did not see anything on xfs. To confirm this absence, I did "grep -in xfs dmesg" and that also came up empty. "grep -in fs dmesg" (no "x") returns type mount and see what the file system is mounted as. The structure needs cleaning in xfs means an xfs_repair is needed. Not sure if this is used by other file systems. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
Hi, I went on the particular server on which the file in question resides. I check the RAID and all the disks show up as OK. I looked in dmesg and did not see anything on xfs. To confirm this absence, I did "grep -in xfs dmesg" and that also came up empty. "grep -in fs dmesg" (no "x") returns 161:checking if image is initramfs... it is 196:usbcore: registered new driver usbfs 256:VFS: Disk quotas dquot_6.5.1 453:EXT3-fs: mounted filesystem with ordered data mode. 494:EXT3 FS on sda1, internal journal 496:EXT3 FS on sda6, internal journal 497:EXT3-fs: mounted filesystem with ordered data mode. 499:EXT3 FS on sda3, internal journal 500:EXT3-fs: mounted filesystem with ordered data mode. 502:EXT3 FS on sda2, internal journal 503:EXT3-fs: mounted filesystem with ordered data mode. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-222B http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 Quoting Joe Landman : On 02/26/2012 12:54 PM, Patrick Haley wrote: Hi, We have recently upgraded our gluster to 3.2.5 and have encountered the following error. Gluster seems somehow confused about one of the files it should be serving up, specifically /projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png If I go to that directory and simply do an ls *.png I get ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning This is usually what happens when you have an underlying xfs file system as your backing store, and have had a failure which has shut the xfs file system down. Have a look in dmesg for the servers, and see if xfs has shut down. Reasons for xfs shutdown include a) bugs in xfs/kernel, b) storage device/RAID failure. Our experience is that "b" happens far more often than "a", though "a" does happen (especially on some kernels). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] "Structure needs cleaning" error
On 02/26/2012 12:54 PM, Patrick Haley wrote: Hi, We have recently upgraded our gluster to 3.2.5 and have encountered the following error. Gluster seems somehow confused about one of the files it should be serving up, specifically /projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png If I go to that directory and simply do an ls *.png I get ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning This is usually what happens when you have an underlying xfs file system as your backing store, and have had a failure which has shut the xfs file system down. Have a look in dmesg for the servers, and see if xfs has shut down. Reasons for xfs shutdown include a) bugs in xfs/kernel, b) storage device/RAID failure. Our experience is that "b" happens far more often than "a", though "a" does happen (especially on some kernels). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] "Structure needs cleaning" error
Hi, We have recently upgraded our gluster to 3.2.5 and have encountered the following error. Gluster seems somehow confused about one of the files it should be serving up, specifically /projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png If I go to that directory and simply do an ls *.png I get ls: BalbacFull_250_200_03Mar_3.png: Structure needs cleaning (along with a listing of the remaining *.png files) I know which brick that file resides and if I go to that brick and look at that file via the underlying nfs, it is fine. I had thought this might be a case of bad meta-pointers, but there are no such pointers on the other bricks. In our gluster log file I see the following error repeated [2012-02-24 13:58:15.507390] E [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-gdata-client-2: remote operation failed: Structure needs cleaning [2012-02-24 13:58:15.507501] W [fuse-bridge.c:184:fuse_entry_cbk] 0-glusterfs-fuse: 176567027: LOOKUP() /projects/philex/PE/2010/Oct18/arch07/BalbacFull_250_200_03Mar_3.png => -1 (Structure needs cleaning) How do we fix this? Especially in general, since I don't know if there are other files in with this error, this is just the only one we have found so far. Currently this error is causing our back-up to fail. Thanks. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: pha...@mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical EngineeringFax:(617) 253-8125 MIT, Room 5-222B http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
