Re: Chasing OOM Issues - good sysctl metrics to use?

[Jan Mikkelsen]

On 10 May 2022, at 10:01, Mark Millard  wrote:
> 
> On 2022-Apr-29, at 13:57, Mark Millard  wrote:
> 
>> On 2022-Apr-29, at 13:41, Pete Wright  wrote:
>>> 
 . . .
>>> 
>>> d'oh - went out for lunch and workstation locked up.  i *knew* i shouldn't 
>>> have said anything lol.
>> 
>> Any interesting console messages ( or dmesg -a or /var/log/messages )?
>> 
> 
> I've been doing some testing of a patch by tijl at FreeBSD.org
> and have reproduced both hang-ups (ZFS/ARC context) and kills
> (UFS/noARC and ZFS/ARC) for "was killed: failed to reclaim
> memory", both with and without the patch. This is with only a
> tiny fraction of the swap partition(s) enabled being put to
> use. So far, the testing was deliberately with
> vm.pageout_oom_seq=12 (the default value). My testing has been
> with main [so: 14].
> 
> But I also learned how to avoid the hang-ups that I got --but
> it costs making kills more likely/quicker, other things being
> equal.
> 
> I discovered that the hang-ups that I got were from all the
> processes that I interact with the system via ending up with
> the process's kernel threads swapped out and were not being
> swapped in. (including sshd, so no new ssh connections). In
> some contexts I only had escaping into the kernel debugger
> available, not even ^T would work. Other times ^T did work.
> 
> So, when I'm willing to risk kills in order to maintain
> the ability to interact normally, I now use in
> /etc/sysctl.conf :
> 
> vm.swap_enabled=0

I have been looking at an OOM related issue. Ignoring the actual leak, the 
problem leads to a process being killed because the system was out of memory. 
This is fine. After that, however, the system console was black with a single 
block cursor and the console keyboard was unresponsive. Caps lock and num lock 
didn’t toggle their lights when pressed.

Using an ssh session, the system looked fine. USB events for the keyboard being 
disconnected and reconnected appeared but the keyboard stayed unresponsive.

Setting vm.swap_enabled=0, as you did above, resolved this problem. After the 
process was killed a perfectly normal console returned.

The interesting thing is that this test system is configured with no swap space.

This is on 13.1-RC5.

> This disables swapping out of process kernel stacks. It
> is just with that option removedfor gaining free RAM, there
> fewer options tried before a kill is initiated. It is not a
> loader-time tunable but is writable, thus the
> /etc/sysctl.conf placement.

Is that really what it does? From a quick look at the code in vm/vm_swapout.c, 
it seems little more complex.

Regards,

Jan M.

Panic in unionfs "it is not a unionfs-vnode"

[Jan Mikkelsen]

Hi,

I got the panic below in unionfs. Head as at 325569, 2017-11-09 12:41:00 +1100 
(Thu, 09 Nov 2017).

This is a ufs filesystem union mounted on top of a read-only ufs /etc. I know 
unionfs has "architecture issues”. Is this resolvable? 

panic: unionfs: it is not unionfs-vnode
cpuid = 5
time = 1510678800
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe02c418c280
vpanic() at vpanic+0x19c/frame 0xfe02c418c300
kassert_panic() at kassert_panic+0x126/frame 0xfe02c418c370
unionfs_lock() at unionfs_lock+0x635/frame 0xfe02c418c400
VOP_LOCK1_APV() at VOP_LOCK1_APV+0xd9/frame 0xfe02c418c430
_vn_lock() at _vn_lock+0x66/frame 0xfe02c418c4a0
vget() at vget+0x82/frame 0xfe02c418c4e0
cache_lookup() at cache_lookup+0x787/frame 0xfe02c418c5c0
vfs_cache_lookup() at vfs_cache_lookup+0xac/frame 0xfe02c418c620
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xd3/frame 0xfe02c418c650
lookup() at lookup+0x682/frame 0xfe02c418c6f0
namei() at namei+0x51a/frame 0xfe02c418c7b0
kern_statat() at kern_statat+0x98/frame 0xfe02c418c9c0
sys_fstatat() at sys_fstatat+0x2f/frame 0xfe02c418cac0
amd64_syscall() at amd64_syscall+0x79b/frame 0xfe02c418cbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfe02c418cbf0
--- syscall (552, FreeBSD ELF64, sys_fstatat), rip = 0x800dd48da, rsp = 
0x7fffddc8, rbp = 0x7fffde70 —


Thanks,

Jan Mikkelsen

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: malloc() call somehow calling the rtld malloc() implementaion

[Jan Mikkelsen]

Hi,

Are you being affected by this rtld behaviour:

https://lists.freebsd.org/pipermail/freebsd-hackers/2016-July/049769.html 


https://lists.freebsd.org/pipermail/freebsd-stable/2015-July/082751.html 


(I still find this a bit scary.)

Regards,

Jan.


> On 27 Jan 2017, at 19:31, Don Lewis  wrote:
> 
> I've been attempting to get OpenOffice to build properly in a
> clang400-import poudriere jail and have run into a mystery.  The build
> procedure creates a c++ executable "idlc", which is used to build other
> stuff.  The new operator has been overrriden to call a custom memory
> allocator, which I have configured to call the system version of
> malloc().
> 
> At some point idlc crashes because it has allocated a 16 byte structure
> and the compiler is using "movaps %xmm0,(%rax)" to initialize it, which
> requires 16 byte alignment.  Unfortunately this structure is only 8 byte
> aligned, causing a bus error.  This shouldn't be happening because our
> system malloc() always seems to do the proper alignment.  It appears
> that intead of calling the version of malloc() in libc, the simple
> version of malloc() built into rtld is being called instead.
> 
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> (gdb) break main
> Breakpoint 1 at 0x43b1f6: file idlcmain.cxx, line 34.
> (gdb) run @/tmp/r
> Starting program: 
> /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/idlc
>  @/tmp/r
> [New LWP 101304]
> [New Thread 802616000 (LWP 101304/idlc)]
> [Switching to Thread 802616000 (LWP 101304/idlc)]
> 
> Breakpoint 1, main (argc=2, argv=0x7fffb278) at idlcmain.cxx:34
> 34SAL_IMPLEMENT_MAIN_WITH_ARGS(argc, argv)
> (gdb) break malloc
> Breakpoint 2 at 0x8006a5f01: file 
> /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c, line 
> 163.
> (gdb) cont
> Continuing.
> Trace 12474/1: "Min Prioriy for policy '2' == '0'
> "
> Trace 12474/1: "Max Prioriy for policy '2' == '103'
> "
> /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/idlc:
>  compiling 1 source files ... 
> Compiling: 
> /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/offapi/com/sun/star/i18n/KParseTokens.idl
> 
> Breakpoint 2, malloc (nbytes=343)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c: 
> No such file or directory.
>   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> Current language:  auto; currently minimal
> (gdb) cont
> Continuing.
> 
> Breakpoint 2, malloc (nbytes=32)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> Trace 13112/2: "ChildStatusProc : starting 
> '/wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/ucpp'"
> [New Thread 802616500 (LWP 100249/idlc)]
> [Switching to Thread 802616500 (LWP 100249/idlc)]
> 
> Breakpoint 2, malloc (nbytes=19)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> 
> Breakpoint 2, malloc (nbytes=34)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> 
> Breakpoint 2, malloc (nbytes=16)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> 
> Breakpoint 2, malloc (nbytes=16)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> 
> Breakpoint 2, malloc (nbytes=16)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> 
> Breakpoint 2, malloc (nbytes=16)
>at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> 163   in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> (gdb) cont
> Continuing.
> sizeof(AstExprValue)=16
> 
> Program 

Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections

[Jan Mikkelsen]

(Copy for the list, wrong from address problem ...)

On 18/09/2013, at 4:22 PM, Konstantin Belousov kostik...@gmail.com wrote:

 On Tue, Sep 17, 2013 at 11:45:19PM +0200, Ed Schouten wrote:
 [...]
 Honestly, I think we can assume we'll never reach the point where all
 the components listed above will properly have all functions
 partitioned over separate compilation units.
 
 I suspect that it would make a lot of sense to at least enable these
 build flags for our core libraries (libc, libc++, libpthread,
 libcompiler_rt, libcxxrt, etc). We could also enable it on
 INTERNALLIBs (libraries that are not installed into /usr/lib), as for
 these libraries, it would of course not come at any cost.
 
 Would that sound okay?
 
 I think this is a wrong direction. First, the split should be done at
 the source level, as it was usually done forever. One of the offender
 there was you, AFAIR.
 
 Second, I would rather see init and devd, and in fact all other statically
 linked binaries from our base system, to become dynamically linked.  At
 least I added a knob for building toolchain dynamic, but avoided the
 fight of making this default.

Why do things by hand when there are good tools? Note ... as it was usually 
done forever doesn't contain a good argument, and compilers and linkers on 
other platforms have been doing it like this for an awfully long time.

Adding the flags has a benefit in the case where there are many functions in a 
source file and minimal cost when everything is perfect. Not having the flags 
means paying a bigger price when things are not perfect. And things are very 
rarely perfect.

Having the structure of your source code driven by link-time considerations 
when there is a choice seems silly to me. Larger source files gives the 
compiler more scope for optimisation, and you can structure the code in a way 
useful to people working in the codebase.

If you have a moral argument about how code should be structured, I think that 
is separate discussion. Adding the flags has a benefit, regardless of how the 
code is structured. I can see all upside, and I am having trouble seeing a 
problem with adding them at all.

On the static linking vs. dynamic linking argument: I am strongly on the static 
linking side. But that is also a different discussion.

Regards,

Jan Mikkelsen
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: -ffunction-sections, -fdata-sections and -Wl,--gc-sections

[Jan Mikkelsen]

On 18/09/2013, at 4:22 PM, Konstantin Belousov kostik...@gmail.com wrote:

 On Tue, Sep 17, 2013 at 11:45:19PM +0200, Ed Schouten wrote:
 [ ... ]
 Honestly, I think we can assume we'll never reach the point where all
 the components listed above will properly have all functions
 partitioned over separate compilation units.
 
 I suspect that it would make a lot of sense to at least enable these
 build flags for our core libraries (libc, libc++, libpthread,
 libcompiler_rt, libcxxrt, etc). We could also enable it on
 INTERNALLIBs (libraries that are not installed into /usr/lib), as for
 these libraries, it would of course not come at any cost.
 
 Would that sound okay?
 
 I think this is a wrong direction. First, the split should be done at
 the source level, as it was usually done forever. One of the offender
 there was you, AFAIR.
 
 Second, I would rather see init and devd, and in fact all other statically
 linked binaries from our base system, to become dynamically linked.  At
 least I added a knob for building toolchain dynamic, but avoided the
 fight of making this default.

Why do things by hand when there are good tools? Note ... as it was usually 
done forever doesn't contain a good argument, and compilers and linkers on 
other platforms have been doing it like this for an awfully long time.

Adding the flags has a benefit in the case where there are many functions in a 
source file and minimal cost when everything is perfect. Not having the flags 
means paying a bigger price when things are not perfect. And things are very 
rarely perfect.

Having the structure of your source code driven by link-time considerations 
when there is a choice seems silly to me. Larger source files gives the 
compiler more scope for optimisation, and you can structure the code in a way 
useful to people working in the codebase.

If you have a moral argument about how code should be structured, I think that 
is separate discussion. Adding the flags has a benefit, regardless of how the 
code is structured. I can see all upside, and I am having trouble seeing a 
problem with adding them at all.

On the static linking vs. dynamic linking argument: I am strongly on the static 
linking side. But that is also a different discussion.

Regards,

Jan Mikkelsen

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: No working IDE in FreeBSD!

[Jan Mikkelsen]

Unix is the original IDE. FreeBSD is a good modern implementation.

If you don't think that's enough, you need to say what you're really looking 
for, rather than just an IDE. It sounds like you're after an editor with more 
general niftiness. Visual Slickedit?

On 23/02/2012, at 10:22 PM, O. Hartmann wrote:

 Several time ago I tried to do some development within an IDE, not even
 for lectural and educational purposes. Since most of our software is
 written in C/C++ and OpenCL, I highly prefered ANJUTA, since this IDE
 was highly customizable, flexible and even FreeBSD's ancient outdated
 version in the ports suited our needs.
 
 Anjuta does not compile anymore for a long time. I do not know why, I
 filed a PR (ports/161494). So I was looking for an alternative.
 
 I looked for some alternatives. The IDE should be configurable to use
 CLANG. ECLIPSE is to large and it does not fit my purpose. I tried
 devel/CodeBlocks, but CodeBlocks is narrowminded in terms of
 configuration of an alternative compiler and I find it really hard and
 not intuitiv to reconfigure the usage of CLANG.
 
 devel/anjuta is broken, so no chance. I also tried KDevelop, since many
 of our Linux based scientists feel good having this very popular IDE,
 but it is marked broken on FreeBSD.
 
 Before I waste more time on searching for a suitable IDE apart ANJUTA,
 I'd like to ask people here what alternative they would suggest if the
 focus is devel/anjuta. Eclipse is no way, KDevelop is broken, CodeBlocks
 is incapable of being easily adapted to CLANG.
 
 Befor people tend to start a flame war: yes, I'm fine with vi and I'm
 also fine with vim/gvim, but our students need to have the opportunity
 to work with an IDE and our projects are partially that large, so an IDE
 is needed.
 
 Thanks a lot for your patience and recommendations in advance.
 
 Oliver
 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org