For gfs, the recommended solution is to periodically run gfs_tool reclaim on your filesystems at a time of your choosing. Depending on the frequency of your deletes, this might be once a day or once a week. The only downside is the during the reclaim operation, the filesystem is locked from other activities. As the reclaim is relatively fast, this doesn't really cause a problem. But scheduling the command to be run during "idle" times of the day will mitigate the impact.
We attempted to come up with a method of doing this automatically, but there are deadlock lock issues between gfs and the vfs layer that prevent it from being implemented. In addition, there is still the issue of when is the right time to do the reclaim, and this would be application specific. So, just run gfs_tool reclaim if your storage is getting consumed by metadata storage. Kevin On Mon, 2008-10-13 at 15:18 -0500, Jason Huddleston wrote: > I've been watching mine do this for about two months now. I think it > started when I upgraded from RHEL 4.5 to 4.6. The app team only has > about 18 gig used on that 1.7TB drive but they create and delete allot > of files because that is the loading area they used when new data > comes in. In the last month I have seen it go up to 70 to 85% used but > it usually comes back down to about 50% within about 24 hours. > Hopefully they will find a fix for this soon. > > --- > Jay > > Shawn Hood wrote: > > I actually just ran the reclaim on a live filesystem and it seems to > > be working okay now. Hopefully this isn't problematic, as a large > > number of operations in the GFS tool suite operate on mounted > > filesystems. > > > > Shawn > > > > On Mon, Oct 13, 2008 at 4:00 PM, Jason Huddleston > > <[EMAIL PROTECTED]> wrote: > > > > > Shawn, > > > I have been seeing the same thing on one of my clusters (shown below) > > > under Red Hat 4.6. I found some details on this under an article on the > > > open-shared root web site > > > (http://www.open-sharedroot.org/faq/troubleshooting-guide/file-systems/gfs/file-system-full) > > > and an article in Red Hat's knowledge base > > > (http://kbase.redhat.com/faq/FAQ_78_10697.shtm). It seems to be a bug in > > > the > > > reclaim of metadata blocks when an inode is released. I saw a patch > > > (bz298931) released for this in the 2.99.10 cluster release notes but it > > > was > > > reverted (bz298931) a few days after it was submitted. The only suggestion > > > that I have gotten back from Red Hat is to shutdown the app so the GFS > > > drives are not being accessed and then run the "gfs_tool reclaim <mount > > > point>" command. > > > > > > [EMAIL PROTECTED] ~]# gfs_tool df /l1load1 > > > /l1load1: > > > SB lock proto = "lock_dlm" > > > SB lock table = "DWCDR_prod:l1load1" > > > SB ondisk format = 1309 > > > SB multihost format = 1401 > > > Block size = 4096 > > > Journals = 20 > > > Resource Groups = 6936 > > > Mounted lock proto = "lock_dlm" > > > Mounted lock table = "DWCDR_prod:l1load1" > > > Mounted host data = "" > > > Journal number = 13 > > > Lock module flags = > > > Local flocks = FALSE > > > Local caching = FALSE > > > Oopses OK = FALSE > > > > > > Type Total Used Free use% > > > ------------------------------------------------------------------------ > > > inodes 155300 155300 0 100% > > > metadata 2016995 675430 1341565 33% > > > data 452302809 331558847 120743962 73% > > > [EMAIL PROTECTED] ~]# df -h /l1load1 > > > Filesystem Size Used Avail Use% Mounted on > > > /dev/mapper/l1load1--vg-l1load1--lv > > > 1.7T 1.3T 468G 74% /l1load1 > > > [EMAIL PROTECTED] ~]# du -sh /l1load1 > > > 18G /l1load1 > > > > > > ---- > > > Jason Huddleston, RHCE > > > ---- > > > PS-USE-Linux > > > Partner Support - Unix Support and Engineering > > > Verizon Information Processing Services > > > > > > > > > > > > Shawn Hood wrote: > > > > > > > Does GFS reserve blocks for the superuser, a la ext3's "Reserved block > > > > count"? I've had a ~1.1TB FS report that it's full with df reporting > > > > ~100GB remaining. > > > > > > > > > > > > > > > -- > > > Linux-cluster mailing list > > > Linux-cluster@redhat.com > > > https://www.redhat.com/mailman/listinfo/linux-cluster > > > > > > > > > > > > > > > > -- > Linux-cluster mailing list > Linux-cluster@redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster@redhat.com https://www.redhat.com/mailman/listinfo/linux-cluster
