On Sat, May 9, 2009 at 6:59 AM, drew wymore <[email protected]> wrote:
> On Sat, May 9, 2009 at 6:50 AM, Hal Pomeranz <[email protected]> wrote: > > > > I ran into a problem on a server I run last night. df was reporting no > > space > > > left and services obviously stopped working. The weird thing is that > when > > I > > > checked all the directories on the mount point they didn't add up to > the > > > space allocated to the partition. It's a 37GB mounted as / with /var > /bin > > > /boot /sbin /usr /dev /media /proc /opt and .sys > > > > > > I removed some un-needed software on the box to get it back up and > > breathing > > > again but I still have no clue how it filled up, so I have no way of > > knowing > > > if it'll happen again. Originally IIRC it was only using about 17GB of > > space > > > as the meat of storage is on /home on a seperate partition. > > > > > > Hitting me with the clue stick would be appreciated or any ideas on > where > > to > > > look would be great. > > > > I'm guessing you have an "open but unlinked" file someplace. In other > > words, there's a process that's writing a big data file, but that file > > was removed. Since the file has been removed ("unlinked") from the file > > system, tools like "du" won't find the space used by this file, which is > > why your file system totals don't match up. However, the operating > system > > is unable to reclaim the space until all processes that have the file > > open have been terminated. > > > > How can you find those processes? "lsof +L1 | sort -nk7" (as root) > > will show you all "files that are open with link count less than 1", > > (i.e. the open but unlinked files) and then sort them by size, so you can > > find the big ones easier. Use the PID in the second column to kill > > the appropriate processes. Note that not all open but unlinked > > files are necessarily a problem. It's my experience that programs > > like VMware and Wine use open but unlinked files a lot. > > > > -- > > Hal Pomeranz, Founder/CEO Deer Run Associates [email protected] > > Network Connectivity and Security, Systems Management, Training > > _______________________________________________ > > PLUG mailing list > > [email protected] > > http://lists.pdxlinux.org/mailman/listinfo/plug > > > > Thanks Hal, > I found some processes related to Apache and MySQL which aren't a problem. > Nothing else showing up that looks like a problem. > > They could become a problem though. If you have a log file filling up a partition, and you delete it, it won't go away til you kill the process (which is what you just learned). I have a trick which works around this. If you blank the file, rather than deleting it, it will free up the space without killing the process. I use echo "" > /var/log/logfile to accomplish this. -wes _______________________________________________ PLUG mailing list [email protected] http://lists.pdxlinux.org/mailman/listinfo/plug
