Re: How to free used Swap-Space? (from errno=8)

[Peter]

On Wed, Sep 23, 2020 at 12:03:32AM +0300, Konstantin Belousov wrote:
! On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote:
! > So what happens then is this:
! > 
! > $ file scc.e
! > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1
! > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1,
! > for FreeBSD 9.3 (903504), stripped
! > 
! > $ ./scc.e
! > ELF interpreter /libexec/ld-elf.so.1 not found, error 8
! > Abort trap
! > 
! > And this will cost about some (hundred?) kB of swapspace every time it
! > happens. And they do not go away again, neither can the concerned jail
! > do fully die again.
! In what sense it 'costs' ?

Well that amount memory gets occupied. Forever, that is, until
poweroff/reset.

! Can you show exact sequence of commands and outputs that demostrate your
! point ?  What type of filesystem the binaries live on ?

Oh, I didn't care. Originally on ZFS. When I tried to reproduce it,
most likely on an NFS-4 share, as I didn't bother to put it anywhere
special.

! I want to reproduce it locally.

Yes that's great! Lets see which info You are lacking.
Here we are now on my desktop box (mostly same machine, same
configuration, i5-3570, 11.4-p3, amd64).

I explicitely removed all the files that do not get installed
when /etc/src.conf contains the "WITHOUT_LIB32=", but I have the
COMPAT_FREEBSD32 still in the kernel.

Now I fetch such an old R9.3/i386 binary from my backups, and
drop it into some NFS filesystem:
(That binary is only 4kB, I just attach it here, if you wanna try
you can straightaway use that one - in normal operation it just
converts some words stdin to stdout).

admin@disp:510:1/ext/Repos$ dir usr2sys
-rwxr-xr-x   1 bin   bin4316 Apr  7  2016 usr2sys
admin@disp:511:1/ext/Repos$ file usr2sys 
usr2sys: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), 
dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 9.3 (903504), 
stripped
admin@disp:513:1/ext/Repos$ mount | grep Repos
edge-e:/ext/Repos on /ext/Repos (nfs, nfsv4acls)
admin@disp:514:1/ext/Repos$ top | cat
Mem: 952M Active, 1687M Inact, 419M Laundry, 4423M Wired, 774M Buf, 348M Free
ARC: 1940M Total, 1378M MFU, 172M MRU, 2492K Anon, 48M Header, 340M Other
 1134M Compressed, 2749M Uncompressed, 2.43:1 Ratio
Swap: 20G Total, 36M Used, 20G Free

As we see, this machine has 8 Gig installed and currently about no swap
used. Now watch what happens:
epos$ ./usr2sys 
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap

admin@disp:519:1/ext/Repos$ for i in `seq 1000`
> do ./usr2sys
> done
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
...

admin@disp:514:1/ext/Repos$ top | cat
Mem: 1010M Active, 1807M Inact, 419M Laundry, 4523M Wired, 774M Buf, 69M Free
ARC: 1940M Total, 1383M MFU, 166M MRU, 2503K Anon, 48M Header, 340M Other
 1134M Compressed, 2750M Uncompressed, 2.43:1 Ratio
Swap: 20G Total, 36M Used, 20G Free

The free memory has already disappeared!

admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done
...

admin@disp:522:1/ext/Repos$ top | cat
Mem: 2154M Active, 78M Inact, 787M Laundry, 4722M Wired, 774M Buf, 89M Free
ARC: 1753M Total, 1273M MFU, 97M MRU, 2653K Anon, 39M Header, 340M Other
 953M Compressed, 2445M Uncompressed, 2.56:1 Ratio
Swap: 20G Total, 358M Used, 20G Free, 1% Inuse

Now the swapspace starts filling.
Lets see if the placement filesystem makes any difference and go onto UFS:

admin@disp:525:1/ext/Repos$ su -
Password:
root@disp:~ # cp /ext/Repos/usr2sys /var
root@disp:~ # dir /var/usr2sys 
-rwxr-xr-x  1 bin  bin  4316 Sep 22 23:55 /var/usr2sys
root@disp:~ # mount | grep /var
/dev/ada0p5 on /var (ufs, local, soft-updates)

admin@disp:527:1/var$ ./usr2sys 
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap

admin@disp:521:1/ext/Repos$ for i in `seq 5000`; do ./usr2sys ; done
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
...

Ahh, that runs a LOT faster now than on the NFS!

admin@disp:529:1/var$ top | cat
Mem: 1546M Active, 67M Inact, 934M Laundry, 5121M Wired, 774M Buf, 161M Free
ARC: 1646M Total, 1159M MFU, 107M MRU, 2686K Anon, 37M Header, 340M Other
 849M Compressed, 2257M Uncompressed, 2.66:1 Ratio
Swap: 20G Total, 1658M Used, 18G Free, 8% Inuse

But memory leakage is similar to worse.

admin@disp:530:1/var$ df tmp
Filesystem1K-blocks   UsedAvail Capacity  Mounted on
zdesk/var/tmp  24747504 231052 24516452 1%/var/tmp
admin@disp:531:1/var$ cp usr2sys tmp
admin@disp:532:1/var$ cd tmp
admin@disp:533:1/var/tmp$ ./usr2sys 
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap
admin@disp:534:1/var/tmp$ for i in `seq 5000`; do ./usr2sys ; done
...

You can see this is now a ZFS, and the behaviour is basically the same:

Mem: 1497M Active, 5292K Inact, 803M Laundry, 5313M Wired, 774M Buf, 212M Free
ARC: 1432M Total, 963M MFU, 105M MRU, 2511K Anon, 21M Header, 341M Other
 650M Compressed, 1482M Uncompressed, 2.28:1 Ratio
Swap: 20G 

Re: How to free used Swap-Space? (from errno=8)

[Konstantin Belousov]

On Wed, Sep 23, 2020 at 12:03:32AM +0300, Konstantin Belousov wrote:
> On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote:
> > So what happens then is this:
> > 
> > $ file scc.e
> > scc.e: ELF 32-bit LSB executable, Intel 80386, version 1
> > (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1,
> > for FreeBSD 9.3 (903504), stripped
> > 
> > $ ./scc.e
> > ELF interpreter /libexec/ld-elf.so.1 not found, error 8
> > Abort trap
> > 
> > And this will cost about some (hundred?) kB of swapspace every time it
> > happens. And they do not go away again, neither can the concerned jail
> > do fully die again.
> In what sense it 'costs' ?
> 
> Can you show exact sequence of commands and outputs that demostrate your
> point ?  What type of filesystem the binaries live on ?
> 
> I want to reproduce it locally.

I suspect that https://reviews.freebsd.org/D26525 should fix it.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: How to free used Swap-Space? (from errno=8)

[Konstantin Belousov]

On Tue, Sep 22, 2020 at 09:11:49PM +0200, Peter wrote:
> So what happens then is this:
> 
> $ file scc.e
> scc.e: ELF 32-bit LSB executable, Intel 80386, version 1
> (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1,
> for FreeBSD 9.3 (903504), stripped
> 
> $ ./scc.e
> ELF interpreter /libexec/ld-elf.so.1 not found, error 8
> Abort trap
> 
> And this will cost about some (hundred?) kB of swapspace every time it
> happens. And they do not go away again, neither can the concerned jail
> do fully die again.
In what sense it 'costs' ?

Can you show exact sequence of commands and outputs that demostrate your
point ?  What type of filesystem the binaries live on ?

I want to reproduce it locally.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: How to free used Swap-Space? (from errno=8)

[Peter]

I think I can reproduce the problem now. See below.

On Tue, Sep 22, 2020 at 02:09:01PM -0400, Mark Johnston wrote:
! On Tue, Sep 22, 2020 at 07:31:07PM +0200, Peter wrote:
! > There is something, and I don't know who owns that:
! > $ vmstat -m | grep shmfd
! > shmfd1314K   -  473  64,256,1024,8192
! > 
! > But that doesn't look big either.
! 
! That is just the amount of kernel memory used to track a set of objects,
! not the actual object sizes.  Unfortunately, in 11 I don't think there's
! any way to enumerate them other than running kgdb and examining the
! shm_dictionary hash table.

One of the owners of this is also postgres (maybe among others).

! I think I see a possible problem in i915, though I'm not sure if you'd
! trigger it just by using vt(4).  It should be fixed in later FreeBSD
! versions, but is still a problem in 11.  Here's a (untested) patch:

Thank You, I'll keep that one in store, just in case.

But now I found something simpler, while tracking error messages that
came into my glance alongside:

When patching to 11.4-p3, I had been reluctant to recompile lib32 and
install that everywhere, and had kicked it off the systems.
And obviousely, I had missed to recompile some of my old self-written
binaries and they were still i386 and were called by various scripts.

So what happens then is this:

$ file scc.e
scc.e: ELF 32-bit LSB executable, Intel 80386, version 1
(FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1,
for FreeBSD 9.3 (903504), stripped

$ ./scc.e
ELF interpreter /libexec/ld-elf.so.1 not found, error 8
Abort trap

And this will cost about some (hundred?) kB of swapspace every time it
happens. And they do not go away again, neither can the concerned jail
do fully die again.

So, maybe, when removing the lib32 & friends from the system, one
must also remove the "options COMPAT_FREEBSD32" from the kernel, so
that it might not try to run that binary, and maybe that would avoid
the issue. (But then, what if one uses lib32 only in *some* jails?
Some evil user in another jail can then bring along an i386 binary
and crash the system by bloating the mem.)

Anyway, my problem is now solved; as I needed these binaries back in
working order anyway. 


regards,
PMc
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"