Re: [ubuntu-uk] df and du give different results
> > > >> Chris, > >> > >> Have you tried df -hi to show the number of free inodes on the system? > >> It should help to reveal whether it's a problem with the number of > >> inodes used or just hidden files or something.. > >> > >> > > > > Hi Lucy, > > > > I think the inode usage is OK. I tried yesterday and it read 8%. > > > > Thanks for the idea though. > > > > Chris > > > > > I could just be slack space. The disk is allocated in clusters of, say, > 4K (though it could be much bigger), > If only 2K of a cluster is used, it still uses 4K of space. If there are > a lot of small files on the drive, the % of slack space is a killer. That's really interesting actually. The web server holds over 18 thousand images, mostly of around a hundred and odd kilobytes. Your explanation would make perfect sense. dumpe2fs reports a block size of 4096. It's not hard to imagine that with that may small files, there will be some loss. By jove you might have cracked it! Thank you Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
>> Chris, >> >> Have you tried df -hi to show the number of free inodes on the system? >> It should help to reveal whether it's a problem with the number of >> inodes used or just hidden files or something.. >> >> > > Hi Lucy, > > I think the inode usage is OK. I tried yesterday and it read 8%. > > Thanks for the idea though. > > Chris > > I could just be slack space. The disk is allocated in clusters of, say, 4K (though it could be much bigger), If only 2K of a cluster is used, it still uses 4K of space. If there are a lot of small files on the drive, the % of slack space is a killer. Also don't forget that IIRC 10% of an ext3 partition is reserved for use by the root user, you can change this on the flay with ext2tune from memory... -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
> Chris, > > Have you tried df -hi to show the number of free inodes on the system? > It should help to reveal whether it's a problem with the number of > inodes used or just hidden files or something.. > Hi Lucy, I think the inode usage is OK. I tried yesterday and it read 8%. Thanks for the idea though. Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
On 04/01/2008, Chris Rowson <[EMAIL PROTECTED]> wrote: > > > > Chris > > > > I just got this response from another list: > > > > The df command will report all the available space on the disk , in other > > words it will report the number of blocks in the "free" list. > > > > The du command gives you and total number of blocks used by the directory > > that is passed to it as a parameter. > > Chris, Have you tried df -hi to show the number of free inodes on the system? It should help to reveal whether it's a problem with the number of inodes used or just hidden files or something.. -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
> > Chris > > I just got this response from another list: > > The df command will report all the available space on the disk , in other > words it will report the number of blocks in the "free" list. > > The du command gives you and total number of blocks used by the directory > that is passed to it as a parameter. > > The difference between the output is because du doesn't take into account > the blocks taken by the directory itself, nor does it count the blocks used > by the "special files" on the filesystem. Things like your device files etc. > > That is why du (in my experience) always reports less than df. > > Which is more or less what Alan said I think! > HTH > > Stu Hi again, Thanks again for looking into it further for me. I don't buy the 'the space is being used by filesystem itself or device files etc' argument in this case though. It might account for some space, but we're talking nearly half of the available filesystem here (about 12 gig or so) just missing to du . I've got another Dapper server sat here too. This one does web-content filtering and caching (squid and dansguardian) for up to 500 clients and is always under a pretty heavy load. Running du and df on that box doesn't show much of a difference in results to be honest. Repeating the test on a CentOS test server running a LAMP environment like the problematic one, again doesn't show anywere near as much as a difference in results either! To be honest, I can't recall being as stuck as this with any problem in the past. I'd have ignored it by now, but the disk is at 98% capacity according to df. Not good (especially when the people I work with have a real downer on anything Linux too...) Cheers Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Chris I just got this response from another list: The df command will report all the available space on the disk , in other words it will report the number of blocks in the "free" list. The du command gives you and total number of blocks used by the directory that is passed to it as a parameter. The difference between the output is because du doesn't take into account the blocks taken by the directory itself, nor does it count the blocks used by the "special files" on the filesystem. Things like your device files etc. That is why du (in my experience) always reports less than df. Which is more or less what Alan said I think! HTH Stu - Original Message From: Chris Rowson <[EMAIL PROTECTED]> To: British Ubuntu Talk Sent: Thursday, 3 January, 2008 8:18:42 PM Subject: Re: [ubuntu-uk] df and du give different results > > Chris > > I have just done some testing as promised with the hidden trash folder on my > ntfs drive and found that both df -h and du -hs /* did not report the > changes in file sizes on the /media volume until I manually deleted the > files from it (therefore it would appear that both commands respond to > changes in hidden files at least when files are deleted from them). However > they were consistently 0.9 Gb adrift in the sizes they were reporting > despite the fact that the folder I was using for testing was a little under > 300 Mb. > > I am going to do some more testing as this happens to have some bearing on > my line of work. I will post back if I come up with anything > significant/relevant/interesting. > > If anyone out there can provide a full explanation of why/how this occurs I > would be very interested. I am sad enough that I find this kind of thing > fascinating :-) > > I hope you resolve your issue soon. > > Stu > Thanks for that Stu, Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/ __ Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
> > Chris > > I have just done some testing as promised with the hidden trash folder on my > ntfs drive and found that both df -h and du -hs /* did not report the > changes in file sizes on the /media volume until I manually deleted the > files from it (therefore it would appear that both commands respond to > changes in hidden files at least when files are deleted from them). However > they were consistently 0.9 Gb adrift in the sizes they were reporting > despite the fact that the folder I was using for testing was a little under > 300 Mb. > > I am going to do some more testing as this happens to have some bearing on > my line of work. I will post back if I come up with anything > significant/relevant/interesting. > > If anyone out there can provide a full explanation of why/how this occurs I > would be very interested. I am sad enough that I find this kind of thing > fascinating :-) > > I hope you resolve your issue soon. > > Stu > Thanks for that Stu, Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Chris I have just done some testing as promised with the hidden trash folder on my ntfs drive and found that both df -h and du -hs /* did not report the changes in file sizes on the /media volume until I manually deleted the files from it (therefore it would appear that both commands respond to changes in hidden files at least when files are deleted from them). However they were consistently 0.9 Gb adrift in the sizes they were reporting despite the fact that the folder I was using for testing was a little under 300 Mb. I am going to do some more testing as this happens to have some bearing on my line of work. I will post back if I come up with anything significant/relevant/interesting. If anyone out there can provide a full explanation of why/how this occurs I would be very interested. I am sad enough that I find this kind of thing fascinating :-) I hope you resolve your issue soon. Stu - Original Message From: Alan Pope <[EMAIL PROTECTED]> To: British Ubuntu Talk Sent: Thursday, 3 January, 2008 6:33:38 PM Subject: Re: [ubuntu-uk] df and du give different results On Thu, 2008-01-03 at 12:49 +, Chris Rowson wrote: > Now that just doesn't add up. > > Please help me :-O > du counts disk space used at a block level and not accurately counted bytes. There is a -b parameter which gives the more realistic apparent size of files. Also du doesn't by default count hidden files. You're also counting /proc and /dev which aren't really directories with files in on disk. Cheers, Al. ___ Support the World Aids Awareness campaign this month with Yahoo! For Good http://uk.promotions.yahoo.com/forgood/-- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
On Thu, 2008-01-03 at 12:49 +, Chris Rowson wrote: > Now that just doesn't add up. > > Please help me :-O > du counts disk space used at a block level and not accurately counted bytes. There is a -b parameter which gives the more realistic apparent size of files. Also du doesn't by default count hidden files. You're also counting /proc and /dev which aren't really directories with files in on disk. Cheers, Al. signature.asc Description: This is a digitally signed message part -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Chris Rowson wrote: > On Jan 3, 2008 3:44 PM, LeeGroups <[EMAIL PROTECTED]> wrote: > >> Concentrate on the DF's results. I can't remember the syntax but you can >> grep for file sizes. >> Start with files over 50M and work downwards. >> I have this trouble with my MythTV box occasionally, it usually a log >> file that explodes in size before the log rotates have time to remove it... >> >> Lee >> > > Hi Lee, > > Are you sure? I thought df only showed free space left on disks, > rather than let you find problems with filesystems on the disks? I've > gone over the man page for df and although it might just be me being > stupid as usual ;-) I can't seem to figure out how I'd run that kind > of search using df. > > You didn't mean du did you? The problem being that they both give > different results. :-( > > Thanks for replying. > > Chris I meant that when DF says there's no room left on your primary drive, bad things start to happen. DU may not add up to what DF says, but if DF says 1% left on a 28GB partition, it's time to investigate what's using the disk space... GREPing for the biggest files usually fixes that fastest... -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
> sudo du -hs /* gives me. > > 3.1M/bin > 9.4M/boot > 0 /cdrom > 172K/dev > 2.6M/etc > 39M /home > 4.0K/initrd > 0 /initrd.img > 76M /lib > 48K /lost+found > 263G/media > 4.0K/mnt > 4.0K/opt > 514M/proc > 20K /root > 8.1M/sbin > 4.0K/srv > 0 /sys > 12K /tmp > 263M/usr > 14G /var > 0 /vmlinuz Are there any hidden directories in the / directory? cos if there are you won't hit them with "sudo du -hs /* ", try doing a "sudo du -hs /.* " too -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
On Jan 3, 2008 3:44 PM, LeeGroups <[EMAIL PROTECTED]> wrote: > Concentrate on the DF's results. I can't remember the syntax but you can > grep for file sizes. > Start with files over 50M and work downwards. > I have this trouble with my MythTV box occasionally, it usually a log > file that explodes in size before the log rotates have time to remove it... > > Lee Hi Lee, Are you sure? I thought df only showed free space left on disks, rather than let you find problems with filesystems on the disks? I've gone over the man page for df and although it might just be me being stupid as usual ;-) I can't seem to figure out how I'd run that kind of search using df. You didn't mean du did you? The problem being that they both give different results. :-( Thanks for replying. Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
On Jan 3, 2008 3:53 PM, Stuart Bird <[EMAIL PROTECTED]> wrote: > > > Chris > > > "The volume at media is a Windows based NTFS" > > So is mine (attached via USB). The hidden trash file is still created on it > though. I have only recently discovered this "feature" so I am not sure why > it happens but it can be annoying. Of course it may not be related to your > issue but it's handy to know. > Hi again Stu, Thanks for replying :-) I suppose if the .trash folder had been created on the attached device, the space would have been lost from the attached device, rather than from my server's hard disk though wouldn't it? I seem to remember having a similar problem with a digital camera, were when I deleted pictures from it in Ubuntu, I never got the disk space back. Like you say, it was caused by a .trash folder. > > "I'd have thought du would show the file if it was just hidden though > wouldn't it ?" > > I would have thought so too. I am not at my home box right now so can't test > it for you. If you haven't resolved the issue by the time I get home I will > gladly run a few tests and see what results I get. > > Stu That'd be cool if you wouldn't mind too much. I have tried looking for the .trash folder on the server here though and couldn't find it. Cheers Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Chris "The volume at media is a Windows based NTFS" So is mine (attached via USB). The hidden trash file is still created on it though. I have only recently discovered this "feature" so I am not sure why it happens but it can be annoying. Of course it may not be related to your issue but it's handy to know. "Oh, I forgot to mention earlier on, the Ubuntu web server is a virtual server running on VMware. I don't suppose that should make any real difference though" I can't think of any, although I don't really use VMware so would have to bow to someone else's greater knowledge on that one. "I'd have thought du would show the file if it was just hidden though wouldn't it ?" I would have thought so too. I am not at my home box right now so can't test it for you. If you haven't resolved the issue by the time I get home I will gladly run a few tests and see what results I get. Stu ___ Support the World Aids Awareness campaign this month with Yahoo! For Good http://uk.promotions.yahoo.com/forgood/-- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Concentrate on the DF's results. I can't remember the syntax but you can grep for file sizes. Start with files over 50M and work downwards. I have this trouble with my MythTV box occasionally, it usually a log file that explodes in size before the log rotates have time to remove it... Lee Chris Rowson wrote: > It's been hours now, and I'm totally and utterly stumped. > > I've used lsof to check whether or not there are any deleted files > still sitting around taking up space, I've run an fsck, rebooted the > server and deleted some logs, but there is still a very large chunk of > hard disk space missing. > > If I've rebooted the server, surely I should have terminated any > processes that had grabbed onto a file keeping it open, and surely > lsof would have seen any deleted ones hanging about wouldn't it? (I > grepped for DEL and deleted). > > The server is a LAMP server running the Gallery2 software and it's > been up for months. I've Googled the answer to this until my eyes are > raw. > > Things shouldn't be this difficult. > > Chris > > -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
It's been hours now, and I'm totally and utterly stumped. I've used lsof to check whether or not there are any deleted files still sitting around taking up space, I've run an fsck, rebooted the server and deleted some logs, but there is still a very large chunk of hard disk space missing. If I've rebooted the server, surely I should have terminated any processes that had grabbed onto a file keeping it open, and surely lsof would have seen any deleted ones hanging about wouldn't it? (I grepped for DEL and deleted). The server is a LAMP server running the Gallery2 software and it's been up for months. I've Googled the answer to this until my eyes are raw. Things shouldn't be this difficult. Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
> > Chris > > Have a poke around for hidden ".trash" folders, particularly on mounted > media such as USB/Firewire hard drives. I have found that Ubuntu has a habit > of creating these on such volumes and they do not appear to get emptied by > the usual processes. I have often thought I had deleted files then realised > that my disc space had not increased as a result, it was always a hidden > trash file that was the culprit. Manually deleting the files from the trash > files usually gets rid of them. > > It may be the large volume you have at /media? that is hiding such a file. > I'm not sure why that "hidden" data wiuld not be seen by df and du though? > > HTH > > Stu Hi Stu, The volume at media is a Windows based NTFS NAS appliance. It's stored in another building and we use it to back up to. I'm not too bothered with the disk usage on that itself, it's more the local drive on the web server which is making me want to stab my own eyes out at the mo!!! :-p Oh, I forgot to mention earlier on, the Ubuntu web server is a virtual server running on VMware. I don't suppose that should make any real difference though I'd have thought du would show the file if it was just hidden though wouldn't it ? Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Chris Have a poke around for hidden ".trash" folders, particularly on mounted media such as USB/Firewire hard drives. I have found that Ubuntu has a habit of creating these on such volumes and they do not appear to get emptied by the usual processes. I have often thought I had deleted files then realised that my disc space had not increased as a result, it was always a hidden trash file that was the culprit. Manually deleting the files from the trash files usually gets rid of them. It may be the large volume you have at /media? that is hiding such a file. I'm not sure why that "hidden" data wiuld not be seen by df and du though? HTH Stu - Original Message From: Chris Rowson <[EMAIL PROTECTED]> To: British Ubuntu Talk Sent: Thursday, 3 January, 2008 12:49:13 PM Subject: [ubuntu-uk] df and du give different results Help! I'm running a Dapper webserver and I'm having terrible problems with du and df giving different results: df -h gives me. FilesystemSize Used Avail Use% Mounted on /dev/mapper/Ubuntu-root 29G 27G 347M 99% / varrun252M 52K 252M 1% /var/run varlock 252M 4.0K 252M 1% /var/lock udev 252M 52K 252M 1% /dev devshm252M 0 252M 0% /dev/shm //172.18.0.13/linuxbackups 1.4T 710G 671G 52% /media/netbackup /dev/sda5 228M 14M 203M 7% /boot sudo du -hs /* gives me. 3.1M/bin 9.4M/boot 0 /cdrom 172K/dev 2.6M/etc 39M /home 4.0K/initrd 0 /initrd.img 76M /lib 48K /lost+found 263G/media 4.0K/mnt 4.0K/opt 514M/proc 20K /root 8.1M/sbin 4.0K/srv 0 /sys 12K /tmp 263M/usr 14G /var 0 /vmlinuz Now that just doesn't add up. I wondered if it might be a problem with open files, so I've tried lsof | grep deleted and lsof | grep DEL They showed Apache2 and MySQL had some files open so I restarted them. This didn't help so I restarted the server. Still no joy!! Please help me :-O Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/ __ Sent from Yahoo! Mail - a smarter inbox http://uk.mail.yahoo.com -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
On 1/3/08, Alec Wright <[EMAIL PROTECTED]> wrote: > Its probably because the filesystem itself takes up some space. Surely not 12 Gig or so though ? Chris -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
Re: [ubuntu-uk] df and du give different results
Its probably because the filesystem itself takes up some space. On 03/01/2008, Chris Rowson <[EMAIL PROTECTED]> wrote: > Help! > > I'm running a Dapper webserver and I'm having terrible problems with > du and df giving different results: > > df -h gives me. > > FilesystemSize Used Avail Use% Mounted on > /dev/mapper/Ubuntu-root >29G 27G 347M 99% / > varrun252M 52K 252M 1% /var/run > varlock 252M 4.0K 252M 1% /var/lock > udev 252M 52K 252M 1% /dev > devshm252M 0 252M 0% /dev/shm > //172.18.0.13/linuxbackups > 1.4T 710G 671G 52% /media/netbackup > /dev/sda5 228M 14M 203M 7% /boot > > > sudo du -hs /* gives me. > > 3.1M/bin > 9.4M/boot > 0 /cdrom > 172K/dev > 2.6M/etc > 39M /home > 4.0K/initrd > 0 /initrd.img > 76M /lib > 48K /lost+found > 263G/media > 4.0K/mnt > 4.0K/opt > 514M/proc > 20K /root > 8.1M/sbin > 4.0K/srv > 0 /sys > 12K /tmp > 263M/usr > 14G /var > 0 /vmlinuz > > Now that just doesn't add up. > > I wondered if it might be a problem with open files, so I've tried > > lsof | grep deleted and lsof | grep DEL > > They showed Apache2 and MySQL had some files open so I restarted them. > This didn't help so I restarted the server. Still no joy!! > > Please help me :-O > > Chris > > -- > ubuntu-uk@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk > https://wiki.kubuntu.org/UKTeam/ > -- ubuntu-uk@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-uk https://wiki.kubuntu.org/UKTeam/
