Can't figure out what's taking up space on /

2021-08-03 Thread Greg Thomas
grits# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/sd0a  986M936M162K   100%/
/dev/sd0k 57.7G   23.7G   31.1G43%/home
/dev/sd0d  3.9G   10.0K3.7G 0%/tmp
/dev/sd0f  5.8G1.1G4.4G21%/usr
/dev/sd0g  986M234M702M25%/usr/X11R6
/dev/sd0h 16.8G   35.5M   15.9G 0%/usr/local
/dev/sd0j  5.8G2.0K5.5G 0%/usr/obj
/dev/sd0i  1.9G2.0K1.8G 0%/usr/src
/dev/sd0e 13.8G   18.8M   13.1G 0%/var
/dev/sd1c  440G305G113G73%/backup

grits# du -xsh /
186M/

I just removed /bsd.sp to free up a little bit of space but I don't
understand the discrepancy between df and du.  How do I troubleshoot
further?

Thanks,
Greg


Re: Can't figure out what's taking up space on /

2021-08-03 Thread Paul de Weerd
df shows you how much data you can write to an fs, while du shows the
disk usage of files it can find.  If it can't find a file (because
it's been deleted), it won't account for it.  But if it's been deleted
and still held open by some process, it would still consume disk
space.

So it looks like a process has a file open on the root filesystem that
has been deleted.  You're looking for a root-owned process that is
(probably) long-running.  My guess the file is in /dev/ (that's my
crystal ball talking though).

Easiest way out is generally to reboot - this stops all processes
(d0h), dus freeing up all the resources they had tied up, including
files that had been deleted from the filesystem.  But going through
your process list to see if you can spot something that may have done
this can be a good learning experience.  In general, base OpenBSD
daemons don't behave this way.

Cheers,

Paul 'WEiRD' de Weerd

On Tue, Aug 03, 2021 at 12:48:42AM -0700, Greg Thomas wrote:
| grits# df -h
| Filesystem SizeUsed   Avail Capacity  Mounted on
| /dev/sd0a  986M936M162K   100%/
| /dev/sd0k 57.7G   23.7G   31.1G43%/home
| /dev/sd0d  3.9G   10.0K3.7G 0%/tmp
| /dev/sd0f  5.8G1.1G4.4G21%/usr
| /dev/sd0g  986M234M702M25%/usr/X11R6
| /dev/sd0h 16.8G   35.5M   15.9G 0%/usr/local
| /dev/sd0j  5.8G2.0K5.5G 0%/usr/obj
| /dev/sd0i  1.9G2.0K1.8G 0%/usr/src
| /dev/sd0e 13.8G   18.8M   13.1G 0%/var
| /dev/sd1c  440G305G113G73%/backup
| 
| grits# du -xsh /
| 186M/
| 
| I just removed /bsd.sp to free up a little bit of space but I don't
| understand the discrepancy between df and du.  How do I troubleshoot
| further?
| 
| Thanks,
| Greg

-- 
>[<++>-]<+++.>+++[<-->-]<.>+++[<+
+++>-]<.>++[<>-]<+.--.[-]
 http://www.weirdnet.nl/ 



Re: Can't figure out what's taking up space on /

2021-08-03 Thread Sebastien Marie
On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> df shows you how much data you can write to an fs, while du shows the
> disk usage of files it can find.  If it can't find a file (because
> it's been deleted), it won't account for it.  But if it's been deleted
> and still held open by some process, it would still consume disk
> space.
> 
> So it looks like a process has a file open on the root filesystem that
> has been deleted.  You're looking for a root-owned process that is
> (probably) long-running.  My guess the file is in /dev/ (that's my
> crystal ball talking though).
> 
> Easiest way out is generally to reboot - this stops all processes
> (d0h), dus freeing up all the resources they had tied up, including
> files that had been deleted from the filesystem.  But going through
> your process list to see if you can spot something that may have done
> this can be a good learning experience.  In general, base OpenBSD
> daemons don't behave this way.

I agree with Paul: you should have a running process which hold
descriptor on unlinked file.

fstat(1) could be used to see list of opened files, and specially
unlinked files:

 INUM   The inode number of the file.  It will be followed by an asterisk
(‘*’) if the inode is unlinked from disk.


$ fstat | grep -F '* -'
[...]
semarie  chrome   537   25 /tmp   48* -rw---   rwp   279793
[...]

here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
inode=48 (unlinked), the file size is 279793 bytes.

-- 
Sebastien Marie



Re: Can't figure out what's taking up space on /

2021-08-03 Thread Ali Farzanrad
I also suspected that it is a filesystem corruption.
Do you have `async` mount option on your root?

Sebastien Marie  wrote:
> On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > df shows you how much data you can write to an fs, while du shows the
> > disk usage of files it can find.  If it can't find a file (because
> > it's been deleted), it won't account for it.  But if it's been deleted
> > and still held open by some process, it would still consume disk
> > space.
> > 
> > So it looks like a process has a file open on the root filesystem that
> > has been deleted.  You're looking for a root-owned process that is
> > (probably) long-running.  My guess the file is in /dev/ (that's my
> > crystal ball talking though).
> > 
> > Easiest way out is generally to reboot - this stops all processes
> > (d0h), dus freeing up all the resources they had tied up, including
> > files that had been deleted from the filesystem.  But going through
> > your process list to see if you can spot something that may have done
> > this can be a good learning experience.  In general, base OpenBSD
> > daemons don't behave this way.
> 
> I agree with Paul: you should have a running process which hold
> descriptor on unlinked file.
> 
> fstat(1) could be used to see list of opened files, and specially
> unlinked files:
> 
>  INUM   The inode number of the file.  It will be followed by an asterisk
> (‘*’) if the inode is unlinked from disk.
> 
> 
> $ fstat | grep -F '* -'
> [...]
> semarie  chrome   537   25 /tmp   48* -rw---   rwp   279793
> [...]
> 
> here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
> inode=48 (unlinked), the file size is 279793 bytes.
> 
> -- 
> Sebastien Marie
> 
> 



Re: Can't figure out what's taking up space on /

2021-08-03 Thread Greg Thomas
I'm definitely suffering from filesystem corruption on root.  I had
rebooted last night with no change.

I have no options for mounting root.

grits# cat /etc/fstab
16a27b4b4549ce04.b none swap sw
16a27b4b4549ce04.a / ffs rw 1 1
16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
/dev/sd1c /backup ffs rw,nodev,nosuid 1 2

I need to upgrade so I can do that from scratch.  This is my backup server
so the configuration is pretty simple.

Not sure fsck output helps here?

grits# fsck /dev/sd0a
** /dev/rsd0a (NO WRITE)
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
fragmentation)

Anyway, I'll reinstall unless someone has more learning experiences for me.

And thank you to Paul for giving a quick explanation of the difference
between df and du.

Thanks all!



On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad 
wrote:

> I also suspected that it is a filesystem corruption.
> Do you have `async` mount option on your root?
>
> Sebastien Marie  wrote:
> > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > > df shows you how much data you can write to an fs, while du shows the
> > > disk usage of files it can find.  If it can't find a file (because
> > > it's been deleted), it won't account for it.  But if it's been deleted
> > > and still held open by some process, it would still consume disk
> > > space.
> > >
> > > So it looks like a process has a file open on the root filesystem that
> > > has been deleted.  You're looking for a root-owned process that is
> > > (probably) long-running.  My guess the file is in /dev/ (that's my
> > > crystal ball talking though).
> > >
> > > Easiest way out is generally to reboot - this stops all processes
> > > (d0h), dus freeing up all the resources they had tied up, including
> > > files that had been deleted from the filesystem.  But going through
> > > your process list to see if you can spot something that may have done
> > > this can be a good learning experience.  In general, base OpenBSD
> > > daemons don't behave this way.
> >
> > I agree with Paul: you should have a running process which hold
> > descriptor on unlinked file.
> >
> > fstat(1) could be used to see list of opened files, and specially
> > unlinked files:
> >
> >  INUM   The inode number of the file.  It will be followed by an
> asterisk
> > (‘*’) if the inode is unlinked from disk.
> >
> >
> > $ fstat | grep -F '* -'
> > [...]
> > semarie  chrome   537   25 /tmp   48* -rw---   rwp
>  279793
> > [...]
> >
> > here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
> > inode=48 (unlinked), the file size is 279793 bytes.
> >
> > --
> > Sebastien Marie
> >
> >
>
>


Re: Can't figure out what's taking up space on /

2021-08-03 Thread Otto Moerbeek
On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:

> I'm definitely suffering from filesystem corruption on root.  I had
> rebooted last night with no change.
> 
> I have no options for mounting root.
> 
> grits# cat /etc/fstab
> 16a27b4b4549ce04.b none swap sw
> 16a27b4b4549ce04.a / ffs rw 1 1
> 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
> 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
> 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
> 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
> 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
> 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
> 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
> 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
> /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
> 
> I need to upgrade so I can do that from scratch.  This is my backup server
> so the configuration is pretty simple.
> 
> Not sure fsck output helps here?
> 
> grits# fsck /dev/sd0a
> ** /dev/rsd0a (NO WRITE)
> ** Last Mounted on /
> ** Root file system
> ** Phase 1 - Check Blocks and Sizes
> ** Phase 2 - Check Pathnames
> ** Phase 3 - Check Connectivity
> ** Phase 4 - Check Reference Counts
> ** Phase 5 - Check Cyl groups
> 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
> fragmentation)
> 
> Anyway, I'll reinstall unless someone has more learning experiences for me.
> 
> And thank you to Paul for giving a quick explanation of the difference
> between df and du.
> 
> Thanks all!

fsck looks normal for a mounted filesystem.

but did you try following Paul's advice to find an open file that has
no directory entry? That is not corruption, but explains why more
storage is in use than du shows.

-Otto

> 
> 
> 
> On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad 
> wrote:
> 
> > I also suspected that it is a filesystem corruption.
> > Do you have `async` mount option on your root?
> >
> > Sebastien Marie  wrote:
> > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > > > df shows you how much data you can write to an fs, while du shows the
> > > > disk usage of files it can find.  If it can't find a file (because
> > > > it's been deleted), it won't account for it.  But if it's been deleted
> > > > and still held open by some process, it would still consume disk
> > > > space.
> > > >
> > > > So it looks like a process has a file open on the root filesystem that
> > > > has been deleted.  You're looking for a root-owned process that is
> > > > (probably) long-running.  My guess the file is in /dev/ (that's my
> > > > crystal ball talking though).
> > > >
> > > > Easiest way out is generally to reboot - this stops all processes
> > > > (d0h), dus freeing up all the resources they had tied up, including
> > > > files that had been deleted from the filesystem.  But going through
> > > > your process list to see if you can spot something that may have done
> > > > this can be a good learning experience.  In general, base OpenBSD
> > > > daemons don't behave this way.
> > >
> > > I agree with Paul: you should have a running process which hold
> > > descriptor on unlinked file.
> > >
> > > fstat(1) could be used to see list of opened files, and specially
> > > unlinked files:
> > >
> > >  INUM   The inode number of the file.  It will be followed by an
> > asterisk
> > > (‘*’) if the inode is unlinked from disk.
> > >
> > >
> > > $ fstat | grep -F '* -'
> > > [...]
> > > semarie  chrome   537   25 /tmp   48* -rw---   rwp
> >  279793
> > > [...]
> > >
> > > here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
> > > inode=48 (unlinked), the file size is 279793 bytes.
> > >
> > > --
> > > Sebastien Marie
> > >
> > >
> >
> >



Re: Can't figure out what's taking up space on /

2021-08-03 Thread Greg Thomas
I thought Paul's advice only applies if I was trying to figure it out
before rebooting?  I'd already rebooted before sending my first email.



On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek  wrote:

> On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
>
> > I'm definitely suffering from filesystem corruption on root.  I had
> > rebooted last night with no change.
> >
> > I have no options for mounting root.
> >
> > grits# cat /etc/fstab
> > 16a27b4b4549ce04.b none swap sw
> > 16a27b4b4549ce04.a / ffs rw 1 1
> > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
> > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
> > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
> > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
> > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
> > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
> > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
> > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
> > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
> >
> > I need to upgrade so I can do that from scratch.  This is my backup
> server
> > so the configuration is pretty simple.
> >
> > Not sure fsck output helps here?
> >
> > grits# fsck /dev/sd0a
> > ** /dev/rsd0a (NO WRITE)
> > ** Last Mounted on /
> > ** Root file system
> > ** Phase 1 - Check Blocks and Sizes
> > ** Phase 2 - Check Pathnames
> > ** Phase 3 - Check Connectivity
> > ** Phase 4 - Check Reference Counts
> > ** Phase 5 - Check Cyl groups
> > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
> > fragmentation)
> >
> > Anyway, I'll reinstall unless someone has more learning experiences for
> me.
> >
> > And thank you to Paul for giving a quick explanation of the difference
> > between df and du.
> >
> > Thanks all!
>
> fsck looks normal for a mounted filesystem.
>
> but did you try following Paul's advice to find an open file that has
> no directory entry? That is not corruption, but explains why more
> storage is in use than du shows.
>
> -Otto
>
> >
> >
> >
> > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad 
> > wrote:
> >
> > > I also suspected that it is a filesystem corruption.
> > > Do you have `async` mount option on your root?
> > >
> > > Sebastien Marie  wrote:
> > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > > > > df shows you how much data you can write to an fs, while du shows
> the
> > > > > disk usage of files it can find.  If it can't find a file (because
> > > > > it's been deleted), it won't account for it.  But if it's been
> deleted
> > > > > and still held open by some process, it would still consume disk
> > > > > space.
> > > > >
> > > > > So it looks like a process has a file open on the root filesystem
> that
> > > > > has been deleted.  You're looking for a root-owned process that is
> > > > > (probably) long-running.  My guess the file is in /dev/ (that's my
> > > > > crystal ball talking though).
> > > > >
> > > > > Easiest way out is generally to reboot - this stops all processes
> > > > > (d0h), dus freeing up all the resources they had tied up, including
> > > > > files that had been deleted from the filesystem.  But going through
> > > > > your process list to see if you can spot something that may have
> done
> > > > > this can be a good learning experience.  In general, base OpenBSD
> > > > > daemons don't behave this way.
> > > >
> > > > I agree with Paul: you should have a running process which hold
> > > > descriptor on unlinked file.
> > > >
> > > > fstat(1) could be used to see list of opened files, and specially
> > > > unlinked files:
> > > >
> > > >  INUM   The inode number of the file.  It will be followed by an
> > > asterisk
> > > > (‘*’) if the inode is unlinked from disk.
> > > >
> > > >
> > > > $ fstat | grep -F '* -'
> > > > [...]
> > > > semarie  chrome   537   25 /tmp   48* -rw---   rwp
> > >  279793
> > > > [...]
> > > >
> > > > here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
> > > > inode=48 (unlinked), the file size is 279793 bytes.
> > > >
> > > > --
> > > > Sebastien Marie
> > > >
> > > >
> > >
> > >
>


Re: Can't figure out what's taking up space on /

2021-08-03 Thread Otto Moerbeek
On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote:

> I thought Paul's advice only applies if I was trying to figure it out
> before rebooting?  I'd already rebooted before sending my first email.

OK, did the free space come back in df after reboot? If so, then it's
programs having open files that are unlinked for sure.

-Otto

> 
> 
> 
> On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek  wrote:
> 
> > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
> >
> > > I'm definitely suffering from filesystem corruption on root.  I had
> > > rebooted last night with no change.
> > >
> > > I have no options for mounting root.
> > >
> > > grits# cat /etc/fstab
> > > 16a27b4b4549ce04.b none swap sw
> > > 16a27b4b4549ce04.a / ffs rw 1 1
> > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
> > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
> > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
> > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
> > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
> > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
> > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
> > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
> > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
> > >
> > > I need to upgrade so I can do that from scratch.  This is my backup
> > server
> > > so the configuration is pretty simple.
> > >
> > > Not sure fsck output helps here?
> > >
> > > grits# fsck /dev/sd0a
> > > ** /dev/rsd0a (NO WRITE)
> > > ** Last Mounted on /
> > > ** Root file system
> > > ** Phase 1 - Check Blocks and Sizes
> > > ** Phase 2 - Check Pathnames
> > > ** Phase 3 - Check Connectivity
> > > ** Phase 4 - Check Reference Counts
> > > ** Phase 5 - Check Cyl groups
> > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
> > > fragmentation)
> > >
> > > Anyway, I'll reinstall unless someone has more learning experiences for
> > me.
> > >
> > > And thank you to Paul for giving a quick explanation of the difference
> > > between df and du.
> > >
> > > Thanks all!
> >
> > fsck looks normal for a mounted filesystem.
> >
> > but did you try following Paul's advice to find an open file that has
> > no directory entry? That is not corruption, but explains why more
> > storage is in use than du shows.
> >
> > -Otto
> >
> > >
> > >
> > >
> > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad 
> > > wrote:
> > >
> > > > I also suspected that it is a filesystem corruption.
> > > > Do you have `async` mount option on your root?
> > > >
> > > > Sebastien Marie  wrote:
> > > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > > > > > df shows you how much data you can write to an fs, while du shows
> > the
> > > > > > disk usage of files it can find.  If it can't find a file (because
> > > > > > it's been deleted), it won't account for it.  But if it's been
> > deleted
> > > > > > and still held open by some process, it would still consume disk
> > > > > > space.
> > > > > >
> > > > > > So it looks like a process has a file open on the root filesystem
> > that
> > > > > > has been deleted.  You're looking for a root-owned process that is
> > > > > > (probably) long-running.  My guess the file is in /dev/ (that's my
> > > > > > crystal ball talking though).
> > > > > >
> > > > > > Easiest way out is generally to reboot - this stops all processes
> > > > > > (d0h), dus freeing up all the resources they had tied up, including
> > > > > > files that had been deleted from the filesystem.  But going through
> > > > > > your process list to see if you can spot something that may have
> > done
> > > > > > this can be a good learning experience.  In general, base OpenBSD
> > > > > > daemons don't behave this way.
> > > > >
> > > > > I agree with Paul: you should have a running process which hold
> > > > > descriptor on unlinked file.
> > > > >
> > > > > fstat(1) could be used to see list of opened files, and specially
> > > > > unlinked files:
> > > > >
> > > > >  INUM   The inode number of the file.  It will be followed by an
> > > > asterisk
> > > > > (‘*’) if the inode is unlinked from disk.
> > > > >
> > > > >
> > > > > $ fstat | grep -F '* -'
> > > > > [...]
> > > > > semarie  chrome   537   25 /tmp   48* -rw---   rwp
> > > >  279793
> > > > > [...]
> > > > >
> > > > > here, chrome (pid 537) has descriptor 25 opened to a file on /tmp
> > > > > inode=48 (unlinked), the file size is 279793 bytes.
> > > > >
> > > > > --
> > > > > Sebastien Marie
> > > > >
> > > > >
> > > >
> > > >
> >



Re: Can't figure out what's taking up space on /

2021-08-04 Thread Greg Thomas
No, after the reboot I'm still in the same situation.  As mentioned earlier
I deleted /bsd.sp so I have a little more free space.

grits# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/sd0a  986M916M   20.1M98%/
/dev/sd0k 57.7G   25.9G   29.0G47%/home
/dev/sd0d  3.9G282K3.7G 0%/tmp
/dev/sd0f  5.8G1.1G4.4G21%/usr
/dev/sd0g  986M234M702M25%/usr/X11R6
/dev/sd0h 16.8G   35.5M   15.9G 0%/usr/local
/dev/sd0j  5.8G2.0K5.5G 0%/usr/obj
/dev/sd0i  1.9G2.0K1.8G 0%/usr/src
/dev/sd0e 13.8G   19.0M   13.1G 0%/var
/dev/sd1c  440G306G112G73%/backup

grits# du -xsh /
166M/

I take it I'm dealing with filesystem corruption as Ali mentioned earlier?

On Tue, Aug 3, 2021 at 11:10 PM Otto Moerbeek  wrote:

> On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote:
>
> > I thought Paul's advice only applies if I was trying to figure it out
> > before rebooting?  I'd already rebooted before sending my first email.
>
> OK, did the free space come back in df after reboot? If so, then it's
> programs having open files that are unlinked for sure.
>
> -Otto
>
> >
> >
> >
> > On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek  wrote:
> >
> > > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
> > >
> > > > I'm definitely suffering from filesystem corruption on root.  I had
> > > > rebooted last night with no change.
> > > >
> > > > I have no options for mounting root.
> > > >
> > > > grits# cat /etc/fstab
> > > > 16a27b4b4549ce04.b none swap sw
> > > > 16a27b4b4549ce04.a / ffs rw 1 1
> > > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
> > > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
> > > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
> > > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
> > > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
> > > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
> > > >
> > > > I need to upgrade so I can do that from scratch.  This is my backup
> > > server
> > > > so the configuration is pretty simple.
> > > >
> > > > Not sure fsck output helps here?
> > > >
> > > > grits# fsck /dev/sd0a
> > > > ** /dev/rsd0a (NO WRITE)
> > > > ** Last Mounted on /
> > > > ** Root file system
> > > > ** Phase 1 - Check Blocks and Sizes
> > > > ** Phase 2 - Check Pathnames
> > > > ** Phase 3 - Check Connectivity
> > > > ** Phase 4 - Check Reference Counts
> > > > ** Phase 5 - Check Cyl groups
> > > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
> > > > fragmentation)
> > > >
> > > > Anyway, I'll reinstall unless someone has more learning experiences
> for
> > > me.
> > > >
> > > > And thank you to Paul for giving a quick explanation of the
> difference
> > > > between df and du.
> > > >
> > > > Thanks all!
> > >
> > > fsck looks normal for a mounted filesystem.
> > >
> > > but did you try following Paul's advice to find an open file that has
> > > no directory entry? That is not corruption, but explains why more
> > > storage is in use than du shows.
> > >
> > > -Otto
> > >
> > > >
> > > >
> > > >
> > > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad <
> ali_farzan...@riseup.net>
> > > > wrote:
> > > >
> > > > > I also suspected that it is a filesystem corruption.
> > > > > Do you have `async` mount option on your root?
> > > > >
> > > > > Sebastien Marie  wrote:
> > > > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
> > > > > > > df shows you how much data you can write to an fs, while du
> shows
> > > the
> > > > > > > disk usage of files it can find.  If it can't find a file
> (because
> > > > > > > it's been deleted), it won't account for it.  But if it's been
> > > deleted
> > > > > > > and still held open by some process, it would still consume
> disk
> > > > > > > space.
> > > > > > >
> > > > > > > So it looks like a process has a file open on the root
> filesystem
> > > that
> > > > > > > has been deleted.  You're looking for a root-owned process
> that is
> > > > > > > (probably) long-running.  My guess the file is in /dev/
> (that's my
> > > > > > > crystal ball talking though).
> > > > > > >
> > > > > > > Easiest way out is generally to reboot - this stops all
> processes
> > > > > > > (d0h), dus freeing up all the resources they had tied up,
> including
> > > > > > > files that had been deleted from the filesystem.  But going
> through
> > > > > > > your process list to see if you can spot something that may
> have
> > > done
> > > > > > > this can be a good learning experience.  In general, base
> OpenBSD
> > > > > > > daemons don't behave this way.
> > > > > >
> > > > > > I agree with Paul: you should have a running process which hold
> > > > > > descriptor o

Re: Can't figure out what's taking up space on /

2021-08-04 Thread Paul de Weerd
On Wed, Aug 04, 2021 at 12:56:57AM -0700, Greg Thomas wrote:
| I take it I'm dealing with filesystem corruption as Ali mentioned earlier?

Could be.  Boot the system in single user mode or the bsd.rd
installation kernel (at the boot prompt type either 'boot -s' or 'boot
bsd.rd').  Enter the shell and run `fsck /`.

However, my next guess is that you have some data stored "under" a
mountpoint somewhere.  Here's what I mean:

# mkdir /mnt/test
# du -sh install69.iso 
544Minstall69.iso
# cp install69.iso /mnt/test
# du -xsh /mnt
545M/mnt
# vnconfig vnd0 /mnt/test/install69.iso
# mount /dev/vnd0c /mnt/test/
# du -xsh /mnt
8.0K /mnt

Since du can't traverse the hierarchy that the install69.iso image has
been mounted over, it also cannot report on the diskspace used by
files in that hierarchy.

Again, boot into single user mode (or from bsd.rd) and figure this
out.

Cheers,

Paul 'WEiRD' de Weerd

| On Tue, Aug 3, 2021 at 11:10 PM Otto Moerbeek  wrote:
| 
| > On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote:
| >
| > > I thought Paul's advice only applies if I was trying to figure it out
| > > before rebooting?  I'd already rebooted before sending my first email.
| >
| > OK, did the free space come back in df after reboot? If so, then it's
| > programs having open files that are unlinked for sure.
| >
| > -Otto
| >
| > >
| > >
| > >
| > > On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek  wrote:
| > >
| > > > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
| > > >
| > > > > I'm definitely suffering from filesystem corruption on root.  I had
| > > > > rebooted last night with no change.
| > > > >
| > > > > I have no options for mounting root.
| > > > >
| > > > > grits# cat /etc/fstab
| > > > > 16a27b4b4549ce04.b none swap sw
| > > > > 16a27b4b4549ce04.a / ffs rw 1 1
| > > > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
| > > > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
| > > > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
| > > > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
| > > > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
| > > > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
| > > > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
| > > > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
| > > > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
| > > > >
| > > > > I need to upgrade so I can do that from scratch.  This is my backup
| > > > server
| > > > > so the configuration is pretty simple.
| > > > >
| > > > > Not sure fsck output helps here?
| > > > >
| > > > > grits# fsck /dev/sd0a
| > > > > ** /dev/rsd0a (NO WRITE)
| > > > > ** Last Mounted on /
| > > > > ** Root file system
| > > > > ** Phase 1 - Check Blocks and Sizes
| > > > > ** Phase 2 - Check Pathnames
| > > > > ** Phase 3 - Check Connectivity
| > > > > ** Phase 4 - Check Reference Counts
| > > > > ** Phase 5 - Check Cyl groups
| > > > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
| > > > > fragmentation)
| > > > >
| > > > > Anyway, I'll reinstall unless someone has more learning experiences
| > for
| > > > me.
| > > > >
| > > > > And thank you to Paul for giving a quick explanation of the
| > difference
| > > > > between df and du.
| > > > >
| > > > > Thanks all!
| > > >
| > > > fsck looks normal for a mounted filesystem.
| > > >
| > > > but did you try following Paul's advice to find an open file that has
| > > > no directory entry? That is not corruption, but explains why more
| > > > storage is in use than du shows.
| > > >
| > > > -Otto
| > > >
| > > > >
| > > > >
| > > > >
| > > > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad <
| > ali_farzan...@riseup.net>
| > > > > wrote:
| > > > >
| > > > > > I also suspected that it is a filesystem corruption.
| > > > > > Do you have `async` mount option on your root?
| > > > > >
| > > > > > Sebastien Marie  wrote:
| > > > > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd wrote:
| > > > > > > > df shows you how much data you can write to an fs, while du
| > shows
| > > > the
| > > > > > > > disk usage of files it can find.  If it can't find a file
| > (because
| > > > > > > > it's been deleted), it won't account for it.  But if it's been
| > > > deleted
| > > > > > > > and still held open by some process, it would still consume
| > disk
| > > > > > > > space.
| > > > > > > >
| > > > > > > > So it looks like a process has a file open on the root
| > filesystem
| > > > that
| > > > > > > > has been deleted.  You're looking for a root-owned process
| > that is
| > > > > > > > (probably) long-running.  My guess the file is in /dev/
| > (that's my
| > > > > > > > crystal ball talking though).
| > > > > > > >
| > > > > > > > Easiest way out is generally to reboot - this stops all
| > processes
| > > > > > > > (d0h), dus freeing up all the resources they had tied up,
| > including
| > > > > > > > files that had been deleted from the filesystem.  But going
| > through
| > > > > > > > your process list

Re: Can't figure out what's taking up space on /

2021-08-04 Thread Greg Thomas
Will do, but I should add that I have done nothing on this box for a couple
of months.  The day before yesterday I realized that I really needed to
backup my laptop, when I went to run my backup script I discovered that I
couldn't reach this server.  When I went to troubleshoot I couldn't login
so I hard rebooted it.

After running fsck in single user mode and letting it fix things I ended up
being able to log in which is when I discovered that / was full.

Anyway, I'll boot into single user mode later today.  I shouldn't be
troubleshooting things in the middle of the night.  I probably could have
retained some more info about my situation if I had waited til the morning
to troubleshoot the other night.

On Wed, Aug 4, 2021 at 1:14 AM Paul de Weerd  wrote:

> On Wed, Aug 04, 2021 at 12:56:57AM -0700, Greg Thomas wrote:
> | I take it I'm dealing with filesystem corruption as Ali mentioned
> earlier?
>
> Could be.  Boot the system in single user mode or the bsd.rd
> installation kernel (at the boot prompt type either 'boot -s' or 'boot
> bsd.rd').  Enter the shell and run `fsck /`.
>
> However, my next guess is that you have some data stored "under" a
> mountpoint somewhere.  Here's what I mean:
>
> # mkdir /mnt/test
> # du -sh install69.iso
> 544Minstall69.iso
> # cp install69.iso /mnt/test
> # du -xsh /mnt
> 545M/mnt
> # vnconfig vnd0 /mnt/test/install69.iso
> # mount /dev/vnd0c /mnt/test/
> # du -xsh /mnt
> 8.0K /mnt
>
> Since du can't traverse the hierarchy that the install69.iso image has
> been mounted over, it also cannot report on the diskspace used by
> files in that hierarchy.
>
> Again, boot into single user mode (or from bsd.rd) and figure this
> out.
>
> Cheers,
>
> Paul 'WEiRD' de Weerd
>
> | On Tue, Aug 3, 2021 at 11:10 PM Otto Moerbeek  wrote:
> |
> | > On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote:
> | >
> | > > I thought Paul's advice only applies if I was trying to figure it out
> | > > before rebooting?  I'd already rebooted before sending my first
> email.
> | >
> | > OK, did the free space come back in df after reboot? If so, then it's
> | > programs having open files that are unlinked for sure.
> | >
> | > -Otto
> | >
> | > >
> | > >
> | > >
> | > > On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek 
> wrote:
> | > >
> | > > > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
> | > > >
> | > > > > I'm definitely suffering from filesystem corruption on root.  I
> had
> | > > > > rebooted last night with no change.
> | > > > >
> | > > > > I have no options for mounting root.
> | > > > >
> | > > > > grits# cat /etc/fstab
> | > > > > 16a27b4b4549ce04.b none swap sw
> | > > > > 16a27b4b4549ce04.a / ffs rw 1 1
> | > > > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
> | > > > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
> | > > > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
> | > > > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
> | > > > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
> | > > > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
> | > > > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
> | > > > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
> | > > > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
> | > > > >
> | > > > > I need to upgrade so I can do that from scratch.  This is my
> backup
> | > > > server
> | > > > > so the configuration is pretty simple.
> | > > > >
> | > > > > Not sure fsck output helps here?
> | > > > >
> | > > > > grits# fsck /dev/sd0a
> | > > > > ** /dev/rsd0a (NO WRITE)
> | > > > > ** Last Mounted on /
> | > > > > ** Root file system
> | > > > > ** Phase 1 - Check Blocks and Sizes
> | > > > > ** Phase 2 - Check Pathnames
> | > > > > ** Phase 3 - Check Connectivity
> | > > > > ** Phase 4 - Check Reference Counts
> | > > > > ** Phase 5 - Check Cyl groups
> | > > > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks, 0.0%
> | > > > > fragmentation)
> | > > > >
> | > > > > Anyway, I'll reinstall unless someone has more learning
> experiences
> | > for
> | > > > me.
> | > > > >
> | > > > > And thank you to Paul for giving a quick explanation of the
> | > difference
> | > > > > between df and du.
> | > > > >
> | > > > > Thanks all!
> | > > >
> | > > > fsck looks normal for a mounted filesystem.
> | > > >
> | > > > but did you try following Paul's advice to find an open file that
> has
> | > > > no directory entry? That is not corruption, but explains why more
> | > > > storage is in use than du shows.
> | > > >
> | > > > -Otto
> | > > >
> | > > > >
> | > > > >
> | > > > >
> | > > > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad <
> | > ali_farzan...@riseup.net>
> | > > > > wrote:
> | > > > >
> | > > > > > I also suspected that it is a filesystem corruption.
> | > > > > > Do you have `async` mount option on your root?
> | > > > > >
> | > > > > > Sebastien Marie  wrote:
> | > > > > > > On Tue, Aug 03, 2021 at 10:03:44AM +0200, Paul de Weerd
> wrote:
> | > > > > > > > df shows

Re: Can't figure out what's taking up space on /

2021-08-04 Thread Greg Thomas
I'm at a loss, I booted in single user mode, ran fsck on /dev/sd0a and it
shows clean.  I still have a large discrepancy between df and du.



On Wed, Aug 4, 2021 at 2:45 AM Greg Thomas 
wrote:

> Will do, but I should add that I have done nothing on this box for a
> couple of months.  The day before yesterday I realized that I really needed
> to backup my laptop, when I went to run my backup script I discovered that
> I couldn't reach this server.  When I went to troubleshoot I couldn't login
> so I hard rebooted it.
>
> After running fsck in single user mode and letting it fix things I ended
> up being able to log in which is when I discovered that / was full.
>
> Anyway, I'll boot into single user mode later today.  I shouldn't be
> troubleshooting things in the middle of the night.  I probably could have
> retained some more info about my situation if I had waited til the morning
> to troubleshoot the other night.
>
> On Wed, Aug 4, 2021 at 1:14 AM Paul de Weerd  wrote:
>
>> On Wed, Aug 04, 2021 at 12:56:57AM -0700, Greg Thomas wrote:
>> | I take it I'm dealing with filesystem corruption as Ali mentioned
>> earlier?
>>
>> Could be.  Boot the system in single user mode or the bsd.rd
>> installation kernel (at the boot prompt type either 'boot -s' or 'boot
>> bsd.rd').  Enter the shell and run `fsck /`.
>>
>> However, my next guess is that you have some data stored "under" a
>> mountpoint somewhere.  Here's what I mean:
>>
>> # mkdir /mnt/test
>> # du -sh install69.iso
>> 544Minstall69.iso
>> # cp install69.iso /mnt/test
>> # du -xsh /mnt
>> 545M/mnt
>> # vnconfig vnd0 /mnt/test/install69.iso
>> # mount /dev/vnd0c /mnt/test/
>> # du -xsh /mnt
>> 8.0K /mnt
>>
>> Since du can't traverse the hierarchy that the install69.iso image has
>> been mounted over, it also cannot report on the diskspace used by
>> files in that hierarchy.
>>
>> Again, boot into single user mode (or from bsd.rd) and figure this
>> out.
>>
>> Cheers,
>>
>> Paul 'WEiRD' de Weerd
>>
>> | On Tue, Aug 3, 2021 at 11:10 PM Otto Moerbeek  wrote:
>> |
>> | > On Tue, Aug 03, 2021 at 10:57:42PM -0700, Greg Thomas wrote:
>> | >
>> | > > I thought Paul's advice only applies if I was trying to figure it
>> out
>> | > > before rebooting?  I'd already rebooted before sending my first
>> email.
>> | >
>> | > OK, did the free space come back in df after reboot? If so, then it's
>> | > programs having open files that are unlinked for sure.
>> | >
>> | > -Otto
>> | >
>> | > >
>> | > >
>> | > >
>> | > > On Tue, Aug 3, 2021 at 10:40 PM Otto Moerbeek 
>> wrote:
>> | > >
>> | > > > On Tue, Aug 03, 2021 at 12:39:54PM -0700, Greg Thomas wrote:
>> | > > >
>> | > > > > I'm definitely suffering from filesystem corruption on root.  I
>> had
>> | > > > > rebooted last night with no change.
>> | > > > >
>> | > > > > I have no options for mounting root.
>> | > > > >
>> | > > > > grits# cat /etc/fstab
>> | > > > > 16a27b4b4549ce04.b none swap sw
>> | > > > > 16a27b4b4549ce04.a / ffs rw 1 1
>> | > > > > 16a27b4b4549ce04.k /home ffs rw,nodev,nosuid 1 2
>> | > > > > 16a27b4b4549ce04.d /tmp ffs rw,nodev,nosuid 1 2
>> | > > > > 16a27b4b4549ce04.f /usr ffs rw,nodev 1 2
>> | > > > > 16a27b4b4549ce04.g /usr/X11R6 ffs rw,nodev 1 2
>> | > > > > 16a27b4b4549ce04.h /usr/local ffs rw,wxallowed,nodev 1 2
>> | > > > > 16a27b4b4549ce04.j /usr/obj ffs rw,nodev,nosuid 1 2
>> | > > > > 16a27b4b4549ce04.i /usr/src ffs rw,nodev,nosuid 1 2
>> | > > > > 16a27b4b4549ce04.e /var ffs rw,nodev,nosuid 1 2
>> | > > > > /dev/sd1c /backup ffs rw,nodev,nosuid 1 2
>> | > > > >
>> | > > > > I need to upgrade so I can do that from scratch.  This is my
>> backup
>> | > > > server
>> | > > > > so the configuration is pretty simple.
>> | > > > >
>> | > > > > Not sure fsck output helps here?
>> | > > > >
>> | > > > > grits# fsck /dev/sd0a
>> | > > > > ** /dev/rsd0a (NO WRITE)
>> | > > > > ** Last Mounted on /
>> | > > > > ** Root file system
>> | > > > > ** Phase 1 - Check Blocks and Sizes
>> | > > > > ** Phase 2 - Check Pathnames
>> | > > > > ** Phase 3 - Check Connectivity
>> | > > > > ** Phase 4 - Check Reference Counts
>> | > > > > ** Phase 5 - Check Cyl groups
>> | > > > > 12852 files, 469195 used, 35516 free (44 frags, 4434 blocks,
>> 0.0%
>> | > > > > fragmentation)
>> | > > > >
>> | > > > > Anyway, I'll reinstall unless someone has more learning
>> experiences
>> | > for
>> | > > > me.
>> | > > > >
>> | > > > > And thank you to Paul for giving a quick explanation of the
>> | > difference
>> | > > > > between df and du.
>> | > > > >
>> | > > > > Thanks all!
>> | > > >
>> | > > > fsck looks normal for a mounted filesystem.
>> | > > >
>> | > > > but did you try following Paul's advice to find an open file that
>> has
>> | > > > no directory entry? That is not corruption, but explains why more
>> | > > > storage is in use than du shows.
>> | > > >
>> | > > > -Otto
>> | > > >
>> | > > > >
>> | > > > >
>> | > > > >
>> | > > > > On Tue, Aug 3, 2021 at 11:39 AM Ali Farzanrad <
>> | 

Re: Can't figure out what's taking up space on /

2021-08-04 Thread Todd C . Miller
On Wed, 04 Aug 2021 13:32:54 -0700, Greg Thomas wrote:

> I'm at a loss, I booted in single user mode, ran fsck on /dev/sd0a and it
> shows clean.  I still have a large discrepancy between df and du.

Did you verify that nothing was hiding under the mount points?  For
example, when booted in single user mode with only the root partition
mounted the /tmp, /home, /var, /usr and /backup directories should
be empty.

 - todd



Re: Can't figure out what's taking up space on /

2021-08-04 Thread Greg Thomas
Thank you Todd.  And I'm sorry to Paul for not reading his post more
thoroughly regarding the mount points.

At some point my rsync script ran while /backup wasn't mounted or
something.  The culprit was there.

On Wed, Aug 4, 2021 at 1:41 PM Todd C. Miller  wrote:

> On Wed, 04 Aug 2021 13:32:54 -0700, Greg Thomas wrote:
>
> > I'm at a loss, I booted in single user mode, ran fsck on /dev/sd0a and it
> > shows clean.  I still have a large discrepancy between df and du.
>
> Did you verify that nothing was hiding under the mount points?  For
> example, when booted in single user mode with only the root partition
> mounted the /tmp, /home, /var, /usr and /backup directories should
> be empty.
>
>  - todd
>


Re: Can't figure out what's taking up space on /

2021-08-04 Thread Greg Thomas
So I found an approximate 750MB in a directory under the /backup mount
point.  Removed that and ended up with sane numbers:

grits# df -h
Filesystem SizeUsed   Avail Capacity  Mounted on
/dev/sd0a  986M166M771M18%/
/dev/sd0k 57.7G   25.9G   29.0G47%/home
/dev/sd0d  3.9G   10.0K3.7G 0%/tmp
/dev/sd0f  5.8G1.1G4.4G21%/usr
/dev/sd0g  986M234M702M25%/usr/X11R6
/dev/sd0h 16.8G   35.5M   15.9G 0%/usr/local
/dev/sd0j  5.8G2.0K5.5G 0%/usr/obj
/dev/sd0i  1.9G2.0K1.8G 0%/usr/src
/dev/sd0e 13.8G   18.8M   13.1G 0%/var
/dev/sd1c  440G306G112G73%/backup

grits# du -sxh /
166M/




On Wed, Aug 4, 2021 at 5:30 PM Jay Hart  wrote:

> For those of us following along, could you post the final disk usage?
>
> Thanks,
>
> Jay
>
> > Thank you Todd.  And I'm sorry to Paul for not reading his post more
> > thoroughly regarding the mount points.
> >
> > At some point my rsync script ran while /backup wasn't mounted or
> > something.  The culprit was there.
> >
> > On Wed, Aug 4, 2021 at 1:41 PM Todd C. Miller 
> wrote:
> >
> >> On Wed, 04 Aug 2021 13:32:54 -0700, Greg Thomas wrote:
> >>
> >> > I'm at a loss, I booted in single user mode, ran fsck on /dev/sd0a
> and it
> >> > shows clean.  I still have a large discrepancy between df and du.
> >>
> >> Did you verify that nothing was hiding under the mount points?  For
> >> example, when booted in single user mode with only the root partition
> >> mounted the /tmp, /home, /var, /usr and /backup directories should
> >> be empty.
> >>
> >>  - todd
> >>
> >
>
>
>


Re: Can't figure out what's taking up space on /

2021-08-05 Thread Bastien Durel
Le mercredi 04 août 2021 à 14:20 -0700, Greg Thomas a écrit :
> At some point my rsync script ran while /backup wasn't mounted or
> something.  The culprit was there.

Hello.

I've done that more than once, especially on NFS-mounted backups.

Since then, I put the mount points directories immutable (before mount)

fremen# mkdir /tmp/foo
fremen# chflags schg /tmp/foo
fremen# touch /tmp/foo/bar
touch: /tmp/foo/bar: Operation not permitted
fremen# ls -loa /tmp/foo
total 8
drwxr-xr-x   2 root  wheel  schg 512 Aug  5 11:01 .
drwxrwxrwt  14 root  wheel  -512 Aug  5 11:01 ..
fremen# mount /dev/vnd0a /tmp/foo/ 
fremen# touch /tmp/foo/bar
fremen# ls -lao /tmp/foo/  
total 8
drwxr-xr-x   2 root  wheel  - 512 Aug  5 11:10 .
drwxrwxrwt  14 root  wheel  - 512 Aug  5 11:10 ..
-rw-r--r--   1 root  wheel  -   0 Aug  5 11:10 bar

Regards,

-- 
Bastien



Re: Can't figure out what's taking up space on /

2021-08-09 Thread Harald Dunkel

On 8/5/21 11:13 AM, Bastien Durel wrote:


Since then, I put the mount points directories immutable (before mount)

fremen# mkdir /tmp/foo
fremen# chflags schg /tmp/foo
fremen# touch /tmp/foo/bar
touch: /tmp/foo/bar: Operation not permitted
fremen# ls -loa /tmp/foo
total 8
drwxr-xr-x   2 root  wheel  schg 512 Aug  5 11:01 .
drwxrwxrwt  14 root  wheel  -512 Aug  5 11:01 ..
fremen# mount /dev/vnd0a /tmp/foo/
fremen# touch /tmp/foo/bar
fremen# ls -lao /tmp/foo/
total 8
drwxr-xr-x   2 root  wheel  - 512 Aug  5 11:10 .
drwxrwxrwt  14 root  wheel  - 512 Aug  5 11:10 ..
-rw-r--r--   1 root  wheel  -   0 Aug  5 11:10 bar

Regards,



Cool idea

Harri



Re: Can't figure out what's taking up space on /

2021-08-09 Thread Otto Moerbeek
On Mon, Aug 09, 2021 at 09:51:39AM +0200, Harald Dunkel wrote:

> On 8/5/21 11:13 AM, Bastien Durel wrote:
> > 
> > Since then, I put the mount points directories immutable (before mount)
> > 
> > fremen# mkdir /tmp/foo
> > fremen# chflags schg /tmp/foo
> > fremen# touch /tmp/foo/bar
> > touch: /tmp/foo/bar: Operation not permitted
> > fremen# ls -loa /tmp/foo
> > total 8
> > drwxr-xr-x   2 root  wheel  schg 512 Aug  5 11:01 .
> > drwxrwxrwt  14 root  wheel  -512 Aug  5 11:01 ..
> > fremen# mount /dev/vnd0a /tmp/foo/
> > fremen# touch /tmp/foo/bar
> > fremen# ls -lao /tmp/foo/
> > total 8
> > drwxr-xr-x   2 root  wheel  - 512 Aug  5 11:10 .
> > drwxrwxrwt  14 root  wheel  - 512 Aug  5 11:10 ..
> > -rw-r--r--   1 root  wheel  -   0 Aug  5 11:10 bar
> > 
> > Regards,
> > 
> 
> Cool idea
> 
> Harri
> 

Note that other approaches (like setting permission bits on the dir
before mount) can encounter the CAVEAT in mount(8).

-Otto