On Sat, May 9, 2009 at 9:34 AM, wes <[email protected]> wrote: > On Sat, May 9, 2009 at 6:59 AM, drew wymore <[email protected]> wrote: > > > On Sat, May 9, 2009 at 6:50 AM, Hal Pomeranz <[email protected]> wrote: > > > > > > I ran into a problem on a server I run last night. df was reporting > no > > > space > > > > left and services obviously stopped working. The weird thing is that > > when > > > I > > > > checked all the directories on the mount point they didn't add up to > > the > > > > space allocated to the partition. It's a 37GB mounted as / with /var > > /bin > > > > /boot /sbin /usr /dev /media /proc /opt and .sys > > > > > > > > I removed some un-needed software on the box to get it back up and > > > breathing > > > > again but I still have no clue how it filled up, so I have no way of > > > knowing > > > > if it'll happen again. Originally IIRC it was only using about 17GB > of > > > space > > > > as the meat of storage is on /home on a seperate partition. > > > > > > > > Hitting me with the clue stick would be appreciated or any ideas on > > where > > > to > > > > look would be great. > > > > > > I'm guessing you have an "open but unlinked" file someplace. In other > > > words, there's a process that's writing a big data file, but that file > > > was removed. Since the file has been removed ("unlinked") from the > file > > > system, tools like "du" won't find the space used by this file, which > is > > > why your file system totals don't match up. However, the operating > > system > > > is unable to reclaim the space until all processes that have the file > > > open have been terminated. > > > > > > How can you find those processes? "lsof +L1 | sort -nk7" (as root) > > > will show you all "files that are open with link count less than 1", > > > (i.e. the open but unlinked files) and then sort them by size, so you > can > > > find the big ones easier. Use the PID in the second column to kill > > > the appropriate processes. Note that not all open but unlinked > > > files are necessarily a problem. It's my experience that programs > > > like VMware and Wine use open but unlinked files a lot. > > > > > > -- > > > Hal Pomeranz, Founder/CEO Deer Run Associates > [email protected] > > > Network Connectivity and Security, Systems Management, Training > > > _______________________________________________ > > > PLUG mailing list > > > [email protected] > > > http://lists.pdxlinux.org/mailman/listinfo/plug > > > > > > > Thanks Hal, > > I found some processes related to Apache and MySQL which aren't a > problem. > > Nothing else showing up that looks like a problem. > > > > > They could become a problem though. If you have a log file filling up a > partition, and you delete it, it won't go away til you kill the process > (which is what you just learned). > > I have a trick which works around this. If you blank the file, rather than > deleting it, it will free up the space without killing the process. I use > echo "" > /var/log/logfile to accomplish this. > > -wes > _______________________________________________ > PLUG mailing list > [email protected] > http://lists.pdxlinux.org/mailman/listinfo/plug >
I did find a file in /var/log that was partly to blame, I also removed some un-needed packages. However I'm still trying to figure out where this space went. Tim mentioned the mount issue which might be a clue, I had a mountpoint of /backups pointing to an external USB device which was having issues (turned out to be an issue with the filesystem, I changed from reiser to ext3 and the problem went away) ... so the question I have, my understanding is that if /backups was having issues the parent partition which is / which is a completely different device, it is possible that junk was getting written to / because of the issue with /backups ? If that is indeed the case, I'm wondering where I can find that garbage data which is sucking up space. I have gone through each of the dir's mounted under / and they don't add up to the amount showing up in df ... I have all my configs and required packages that I built from source available to me so starting over wouldn't take very long. I'm just wondering if there is a way to figure this out before going to that extreme. Would the fact that the partition was created with 1k blocks instead of 4k blocks? Thanks, Drew- _______________________________________________ PLUG mailing list [email protected] http://lists.pdxlinux.org/mailman/listinfo/plug
