Re: Filespace exhaustion on '/' partition
On 11/05/2014 22:07, Sven Joachim wrote: A simple mount --bind / /mnt makes all those files visible under /mnt, and you can delete them at your leisure. I mounted the root partition at /mnt/test and used du. server4:/home/ron# du /mnt/test/mnt -hx --max-depth=1 2.7G/mnt/test/mnt/backupserver 0 /mnt/test/mnt/test 2.7G/mnt/test/mnt server4:/home/ron# The 2.7GB is the remnant of a first backup attempt to a new backup server. We backup over NFS, to a server mounted at /mnt/backupserver. During that backup trial, the new backup server was not configured correctly, and this machine had not - actually - seen it, though it had appeared to do so. As a result, this machine tried to do a backup to that destination, /mnt/backupserver, and had - evidently - filled the root partition before complaining about space. I hadn't understood - then - what had happened, and I corrected (only) the missing NFS export and connectivity. That fix meant that the mountpoint no longer 'pointed' to a set of directories on the root partition but - instead - to the NFS export (correctly). A 'proper' backup then succeeded; what I hadn't realised, until now, was that the original set of failed-test directories was still there, filling the partition and, moreover, now invisible. There are some very clever people on this list; how on earth did anyone know that. I need one more piece of advice. I will delete those files but 'I think' I need to keep /mnt/test/mnt - because that's the main server fs /mnt directory /mnt/test/mnt/backupserver - because that's the normal NFS mountpoint for the backups and I think I can delete everything within, and beneath, /mnt/test/mnt/backupserver Does anyone see a problem with what I intend to do? regards, Ron -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/5370d071.9070...@tesco.net
Re: Filespace exhaustion on '/' partition
On 12 May 2014 23:45, Ron Leach ronle...@tesco.net wrote: On 11/05/2014 22:07, Sven Joachim wrote: A simple mount --bind / /mnt makes all those files visible under /mnt, and you can delete them at your leisure. I mounted the root partition at /mnt/test and used du. server4:/home/ron# du /mnt/test/mnt -hx --max-depth=1 2.7G/mnt/test/mnt/backupserver 0 /mnt/test/mnt/test 2.7G/mnt/test/mnt server4:/home/ron# The 2.7GB is the remnant of a first backup attempt to a new backup server. We backup over NFS, to a server mounted at /mnt/backupserver. During that backup trial, the new backup server was not configured correctly, and this machine had not - actually - seen it, though it had appeared to do so. As a result, this machine tried to do a backup to that destination, /mnt/backupserver, and had - evidently - filled the root partition before complaining about space. I hadn't understood - then - what had happened, and I corrected (only) the missing NFS export and connectivity. That fix meant that the mountpoint no longer 'pointed' to a set of directories on the root partition but - instead - to the NFS export (correctly). A 'proper' backup then succeeded; what I hadn't realised, until now, was that the original set of failed-test directories was still there, filling the partition and, moreover, now invisible. Whenever I create a directory that is going to be a mountpoint, I immediately set its immutable attribute to prevent exactly this. Nothing can be written in an immutable directory. chattr must be run by root user. # mkdir mountpoint # chattr -V +i mountpoint -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/CAMPXz=rhsxkrdyn9zaw9jyvn7x1-vl7t061bnfxe5afyyqc...@mail.gmail.com
Re: Filespace exhaustion on '/' partition
On Mon, May 12, 2014 at 02:45:21PM +0100, Ron Leach wrote: I mounted the root partition at /mnt/test and used du. server4:/home/ron# du /mnt/test/mnt -hx --max-depth=1 2.7G/mnt/test/mnt/backupserver 0 /mnt/test/mnt/test 2.7G/mnt/test/mnt server4:/home/ron# The 2.7GB is the remnant of a first backup attempt to a new backup server. We backup over NFS, to a server mounted at /mnt/backupserver. During that backup trial, the new backup server was not configured correctly, and this machine had not - actually - seen it, though it had appeared to do so. As a result, this machine tried to do a backup to that destination, /mnt/backupserver, and had - evidently - filled the root partition before complaining about space. Here's how to avoid that: Mount /mnt/backupserver Make sure your backup system is writing to /mnt/backupserver/specialcodename/hostname/... If your backup system can't see the specialcodename directory, it should not create it, but give an error message instead. (Test this, and write a wrapper if your backup system obstinately decides to create intermediate directories for you.) -dsr- -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140512142646.gs26...@randomstring.org
Re: Filespace exhaustion on '/' partition
On 20140513_0014+1000, David wrote: On 12 May 2014 23:45, Ron Leach ronle...@tesco.net wrote: On 11/05/2014 22:07, Sven Joachim wrote: A simple mount --bind / /mnt makes all those files visible under /mnt, and you can delete them at your leisure. I mounted the root partition at /mnt/test and used du. server4:/home/ron# du /mnt/test/mnt -hx --max-depth=1 2.7G/mnt/test/mnt/backupserver 0 /mnt/test/mnt/test 2.7G/mnt/test/mnt server4:/home/ron# The 2.7GB is the remnant of a first backup attempt to a new backup server. We backup over NFS, to a server mounted at /mnt/backupserver. During that backup trial, the new backup server was not configured correctly, and this machine had not - actually - seen it, though it had appeared to do so. As a result, this machine tried to do a backup to that destination, /mnt/backupserver, and had - evidently - filled the root partition before complaining about space. I hadn't understood - then - what had happened, and I corrected (only) the missing NFS export and connectivity. That fix meant that the mountpoint no longer 'pointed' to a set of directories on the root partition but - instead - to the NFS export (correctly). A 'proper' backup then succeeded; what I hadn't realised, until now, was that the original set of failed-test directories was still there, filling the partition and, moreover, now invisible. Whenever I create a directory that is going to be a mountpoint, I immediately set its immutable attribute to prevent exactly this. Nothing can be written in an immutable directory. chattr must be run by root user. # mkdir mountpoint # chattr -V +i mountpoint The fact that a mountpoint can be mis-interpreted by the system for the root of a directory sub-tree on the parent disk has been around, as a gotcha, pretty much from the beginning of Unix. 'mountpoint' is also the name of a test program that allows a sysadmin to test for this situation. Too much name space overloading here, IMHO, but I can't make a good suggestion as to what to do about it ;-) -- Paul E Condon pecon...@mesanetworks.net -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140512155214.ga32...@big.lan.gnu
Re: Filespace exhaustion on '/' partition
Ron Leach: There are some very clever people on this list; how on earth did anyone know that. Most of us probably learned it the hard way, just like you did. :-) J. -- If I could travel through time I would go back to yesterday and apologise. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html signature.asc Description: Digital signature
Re: SOLVED: Filespace exhaustion on '/' partition
On 12/05/2014 14:45, Ron Leach wrote: and I think I can delete everything within, and beneath, /mnt/test/mnt/backupserver So I did that. server4:/home/ron# df -h FilesystemSize Used Avail Use% Mounted on /dev/md1 2.8G 170M 2.7G 6% / tmpfs 501M 0 501M 0% /lib/init/rw udev 10M 808K 9.3M 8% /dev tmpfs 501M 0 501M 0% /dev/shm /dev/md6 1.8T 1.4T 406G 78% /nfs /dev/sda1 313M 16M 281M 6% /boot /dev/sdb1 313M 16M 281M 6% /boot2 /dev/md5 38G 2.4G 35G 7% /home /dev/md4 949M 4.3M 945M 1% /tmp /dev/md2 9.4G 1.2G 8.2G 13% /usr /dev/md3 4.7G 773M 3.9G 17% /var server4:/home/ron# /dev/md1 looks normal, now. Thanks to everyone with all their insights and subsequent advice for avoiding the problem, especially Sven whose mount command I used to be able to check the partition and remove the files, while the server was still running. Very grateful, Ron -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/5370f0ac.5080...@tesco.net
Re: Filespace exhaustion on '/' partition
On Du, 11 mai 14, 23:07:01, Sven Joachim wrote: The only way that I know of to check that would be to check one filesystem at a time in single user mode, or from a live CD. Huh, why that? A simple mount --bind / /mnt makes all those files visible under /mnt, and you can delete them at your leisure. Nice! Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser Offtopic discussions among Debian users and developers: http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic http://nuvreauspam.ro/gpg-transition.txt signature.asc Description: Digital signature
Filespace exhaustion on '/' partition
List, We seem to have filled the available space on the '/' partition of our NFS server. Because most of the server's variable data is on separate partitions, I'm not sure what I could remove from '/' partition. df shows the problem, and the space available on the other partitions: server4:/# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/md1 2919360 291932436 100% / tmpfs 512856 0512856 0% /lib/init/rw udev 10240 808 9432 8% /dev tmpfs 512856 0512856 0% /dev/shm /dev/md6 1892786624 1467249964 425536660 78% /nfs /dev/sda1 320310 15665287556 6% /boot /dev/sdb1 320310 15665287556 6% /boot2 /dev/md5 39043328 2431240 36612088 7% /home /dev/md4971648 4324967324 1% /tmp /dev/md2 9755264 1241512 8513752 13% /usr /dev/md3 4872448790660 4081788 17% /var server4:/# This is a live server, relied on by several desktop systems and, to a lesser extent, some other servers. The partition exported to the rest of the network is regularly backed up. Am I correct in thinking that I cannot, while running, shrink or grow any of the partitions? Presumably I could do that if the server was offline, perhaps by running a partition editor from a CD or USB stick, maybe? There is a GUI on this system but, aside from that, few if any 'applications'; we do run samba, but not apache, we run exim, and I notice that open office is installed (which will be long out of date, by now, anyway, and I'll remove). Are there any large-ish services that are believed to not always be necessary on a server, and whose removal might release a reasonable amount of space on '/'? regards, Ron -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/536f8a42.90...@tesco.net
Re: Filespace exhaustion on '/' partition
On Du, 11 mai 14, 15:33:38, Ron Leach wrote:
List,
We seem to have filled the available space on the '/' partition of our NFS
server. Because most of the server's variable data is on separate
partitions, I'm not sure what I could remove from '/' partition. df shows
the problem, and the space available on the other partitions:
server4:/# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md1 2919360 291932436 100% /
If Im reading this figures correctly ('df -h' is much nicer) your / has
somewhere near 2,8 GiB and is full. Considering you have separate /usr
(which has only some 1,2 GiB) and /var this sounds fishy.
tmpfs 512856 0512856 0% /lib/init/rw
udev 10240 808 9432 8% /dev
tmpfs 512856 0512856 0% /dev/shm
/dev/md6 1892786624 1467249964 425536660 78% /nfs
/dev/sda1 320310 15665287556 6% /boot
/dev/sdb1 320310 15665287556 6% /boot2
/dev/md5 39043328 2431240 36612088 7% /home
/dev/md4971648 4324967324 1% /tmp
/dev/md2 9755264 1241512 8513752 13% /usr
/dev/md3 4872448790660 4081788 17% /var
server4:/#
This is a live server, relied on by several desktop systems and, to a lesser
extent, some other servers. The partition exported to the rest of the
network is regularly backed up.
Good.
Am I correct in thinking that I cannot, while running, shrink or grow any of
the partitions? Presumably I could do that if the server was offline,
perhaps by running a partition editor from a CD or USB stick, maybe?
2,9 GiB for / with separate /usr and /var should be plenty. I'd suggest
looking into what is using all that space.
There is a GUI on this system but, aside from that, few if any
'applications'; we do run samba, but not apache, we run exim, and I notice
that open office is installed (which will be long out of date, by now,
anyway, and I'll remove).
These should reside in /usr, so in theory wouldn't help much with your
immediate problem.
Are there any large-ish services that are
believed to not always be necessary on a server, and whose removal might
release a reasonable amount of space on '/'?
See if unused Linux images are installed
dpkg -l linux-*
Removing one could already provide some breathing space, but I would
keep at least two around (the one in use, obviously, and next older
one).
You might want to check the output of
du / -hx --max-depth=1
To see where the 2,9 GiB are, but my bets are on /opt ;)
Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser
Offtopic discussions among Debian users and developers:
http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic
http://nuvreauspam.ro/gpg-transition.txt
signature.asc
Description: Digital signature
Re: Filespace exhaustion on '/' partition
On 11/05/2014 16:10, Andrei POPESCU wrote:
On Du, 11 mai 14, 15:33:38, Ron Leach wrote:
We seem to have filled the available space on the '/' partition of our NFS
server. Because most of the server's variable data is on separate
partitions, I'm not sure what I could remove from '/' partition. df shows
the problem, and the space available on the other partitions:
server4:/# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md1 2919360 291932436 100% /
If Im reading this figures correctly ('df -h' is much nicer) your / has
somewhere near 2,8 GiB and is full. Considering you have separate /usr
(which has only some 1,2 GiB) and /var this sounds fishy.
2,9 GiB for / with separate /usr and /var should be plenty. I'd suggest
looking into what is using all that space.
See if unused Linux images are installed
dpkg -l linux-*
Assuming 'un' means not installed, this looks ok:
server4:/# dpkg -l linux-*
Desired=Unknown/Install/Remove/Purge/Hold
|
Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err:
uppercase=bad)
||/ Name VersionDescription
+++-==-==-
un linux-doc-2.6. none (no description available)
un linux-imagenone (no description available)
un linux-image-2. none (no description available)
ii linux-image-2. 2.6.26+17+lenn Linux 2.6 image on
PPro/Celeron/PII/PIII/P4
ii linux-image-2. 2.6.26-25 Linux 2.6.26 image on
PPro/Celeron/PII/PIII/
un linux-initramf none (no description available)
un linux-kernel-l none (no description available)
un linux-latest-m none (no description available)
un linux-modules- none (no description available)
ii linux-sound-ba 1.0.17.dfsg-4 base package for ALSA and OSS sound
systems
server4:/#
You might want to check the output of
du / -hx --max-depth=1
To see where the 2,9 GiB are, but my bets are on /opt ;)
server4:/# du / -hx --max-depth=1
0 /var
0 /nfs
1.0K/boot
1.0K/boot2
0 /home
4.0K/tmp
0 /usr
80M /etc
0 /media
64M /lib
5.0M/sbin
0 /selinux
4.1M/bin
0 /dev
0 /proc
12K /mnt
12M /root
0 /sys
0 /srv
0 /opt
165M/
server4:/#
This doesn't suggest anything like 2.8GB, does it? (du does include
the content of sub-directories in its calculations, doesn't it?)
Your math was fine, but here's df -h for clarity
server4:/# df -h
FilesystemSize Used Avail Use% Mounted on
/dev/md1 2.8G 2.8G 36K 100% /
tmpfs 501M 0 501M 0% /lib/init/rw
udev 10M 808K 9.3M 8% /dev
tmpfs 501M 0 501M 0% /dev/shm
/dev/md6 1.8T 1.4T 406G 78% /nfs
/dev/sda1 313M 16M 281M 6% /boot
/dev/sdb1 313M 16M 281M 6% /boot2
/dev/md5 38G 2.4G 35G 7% /home
/dev/md4 949M 4.3M 945M 1% /tmp
/dev/md2 9.4G 1.2G 8.2G 13% /usr
/dev/md3 4.7G 773M 3.9G 17% /var
server4:/#
md is raid1, and xfs. I tried fsck to see if there was some kind of
problem, but it refused to check an xfs filesystem. xfs_check,
itself, declined to run because /dev/md1 is mounted, and rw.
Running du to two levels reveals largest quantities for:
80M /etc
76M /etc/webmin
11M /root/.thumbnails
But there's nothing remotely approaching 2.8GB .
I'll try and look round the filesystem, I imagine xfs will log
something somewhere if it notices something wrong. I would like to
check the fs; if I recall, the system checks the xfs filesystem during
start up so, perhaps later today, I could take the system offline and
reboot it to force a check.
regards, Ron
--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: https://lists.debian.org/536f9f19.1020...@tesco.net
Re: Filespace exhaustion on '/' partition
On 2014-05-11 18:02 +0200, Ron Leach wrote: server4:/# du / -hx --max-depth=1 0 /var 0 /nfs 1.0K/boot 1.0K/boot2 0 /home 4.0K/tmp 0 /usr 80M /etc 0 /media 64M /lib 5.0M/sbin 0 /selinux 4.1M/bin 0 /dev 0 /proc 12K /mnt 12M /root 0 /sys 0 /srv 0 /opt 165M/ server4:/# This doesn't suggest anything like 2.8GB, does it? (du does include the content of sub-directories in its calculations, doesn't it?) Right, but it does not count unlinked but still open files. I'll try and look round the filesystem, I imagine xfs will log something somewhere if it notices something wrong. I would like to check the fs; if I recall, the system checks the xfs filesystem during start up so, perhaps later today, I could take the system offline and reboot it to force a check. Rebooting also frees up the space of unlinked files, but so does killing the processes which keep them open. On a long running system there can be quite a lot of those files, though: # lsof | grep deleted | wc -l 2058 Cheers, Sven -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/8761lcfg6z@turtle.gmx.de
Re: Filespace exhaustion on '/' partition
On 11/05/2014 17:11, Sven Joachim wrote: # lsof | grep deleted | wc -l # lsof | grep deleted | wc -l 3 # But it took 3 or 4 seconds to count them. :) I was beginning to think, gosh, there must really be a lot of those. But it's a good point, Sven, because there still might be some issue that is keeping even 1 of those in some extended length, but deleted, file. Maybe not, though, on '/'. I'll reboot the system at the end of the day. Though it's a weekend, at the moment it'll be in use for a couple of hours, yet. regards, Ron -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/536fa70c.30...@tesco.net
Re: Filespace exhaustion on '/' partition
Ron Leach ronle...@tesco.net writes: On 11/05/2014 17:11, Sven Joachim wrote: # lsof | grep deleted | wc -l # lsof | grep deleted | wc -l 3 # But it took 3 or 4 seconds to count them. :) I was beginning to think, gosh, there must really be a lot of those. But it's a good point, Sven, because there still might be some issue that is keeping even 1 of those in some extended length, but deleted, file. Maybe not, though, on '/'. I'll reboot the system at the end of the day. Though it's a weekend, at the moment it'll be in use for a couple of hours, yet. Another possibility to look into is that there might be files under your mount points. For example, you might have saved files under /usr on your root filesystem, but later mounted a /usr filesystem on top without deleting the files in the original filesystem. The only way that I know of to check that would be to check one filesystem at a time in single user mode, or from a live CD. -- Carl Johnsonca...@peak.org -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/878uq859ip.fsf@oak.localnet
Re: Filespace exhaustion on '/' partition
On 05/11/2014 04:46 PM, Carl Johnson wrote: Another possibility to look into is that there might be files under your mount points. For example, you might have saved files under /usr on your root filesystem, but later mounted a /usr filesystem on top without deleting the files in the original filesystem. The only way that I know of to check that would be to check one filesystem at a time in single user mode, or from a live CD. That's a good point. You could just mount the rootfs read-only somewhere else. That way you can look at it while the system is still live. - PaulNM -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/536fe44d.2040...@paulscrap.com
Re: Filespace exhaustion on '/' partition
On 2014-05-11 22:46 +0200, Carl Johnson wrote: Ron Leach ronle...@tesco.net writes: On 11/05/2014 17:11, Sven Joachim wrote: # lsof | grep deleted | wc -l # lsof | grep deleted | wc -l 3 # But it took 3 or 4 seconds to count them. :) I was beginning to think, gosh, there must really be a lot of those. But it's a good point, Sven, because there still might be some issue that is keeping even 1 of those in some extended length, but deleted, file. Maybe not, though, on '/'. I'll reboot the system at the end of the day. Though it's a weekend, at the moment it'll be in use for a couple of hours, yet. Another possibility to look into is that there might be files under your mount points. For example, you might have saved files under /usr on your root filesystem, but later mounted a /usr filesystem on top without deleting the files in the original filesystem. Indeed. The only way that I know of to check that would be to check one filesystem at a time in single user mode, or from a live CD. Huh, why that? A simple mount --bind / /mnt makes all those files visible under /mnt, and you can delete them at your leisure. Cheers, Sven -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/87tx8wdnyi@turtle.gmx.de
Re: Filespace exhaustion on '/' partition
On 11/05/2014 22:07, Sven Joachim wrote: On 2014-05-11 22:46 +0200, Carl Johnson wrote: Another possibility to look into is that there might be files under your mount points. For example, you might have saved files under /usr on your root filesystem, but later mounted a /usr filesystem on top without deleting the files in the original filesystem. It's certainly worth checking. All these physical partitions, the raid1 partition pairs, the xfs filesystem, and the mount points were created during the debian-installer's partitioning and filesystems layout. (I think I didn't mention, in this thread, that the server is running Lenny.) Nevertheless, checking to see what is 'really' there, on /dev/md1, is worth doing. Obviously, something is causing df to believe that the mount point is full. A simple mount --bind / /mnt makes all those files visible under /mnt, and you can delete them at your leisure. I'll do this, this way, during tomorrow. I haven't rebooted the system yet, either (I'd wanted to force an xfs_check). I have checked /var/log/messages, though: there's a weekly xfs check, and it passed last weekend. regards, Ron -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/536feac6.8090...@tesco.net
