Re: libc/libsys split coming soon
On 05-02-24 17:02, Brooks Davis wrote:
Could you do a quick test with an exe linked to libsys but not libc running
under Valgrind memcheck, please?
Could you suggest a more concrete example?
This little example seems to be OK
void _start(void)
{
_exit(0);
}
However I do see quite a few new testcase failures. Some are libsys
related and fairly unimportant. There are also a few more serious issues
that I'm still investigating, not necessarily anything to do with libsys.
A+
Paul
Re: sanitizers broken (was RE: libc/libsys split coming soon) [errno in libsys: any analogy to __elf_aux_vector ?]
On Thu, Feb 22, 2024 at 10:16:30AM -0800, Mark Millard wrote: > Brooks Davis wrote on > Date: Thu, 22 Feb 2024 02:03:09 UTC : > > > On Thu, Feb 22, 2024 at 02:57:13AM +0200, Konstantin Belousov wrote: > > > On Wed, Feb 21, 2024 at 08:20:25PM +, Brooks Davis wrote: > > > > That's probably worth a shot. Static linking will work anyway because > > > > libc.a in effect embeds libsys to retain compatability. > > > Please do not add libsys.so to the ABI. Right now it is an implementation > > > detail of libthr and libc, and it is preferable to not change it, at least > > > not yet, and definitely not to solve some minor internal issues. > > > > Indeed, on further reflection I agree. Another option occured to me > > which I intend to persue tomorrow: explicitly link the sanitizer .so > > files with libsys. At least in the base system that should be straight > > foward. > > Does the errno move to libsys have any problems similar to > the __elf_aux_vector move to libsys --that might also lead > to needing -lsys (for things as the are now)? I don't think so. With errno, there is still a copy in libc, it's just not used because the libsys on takes precidence. With __elf_aux_vector we were working around a different issue where the wrong copy was being updated by rtld which I resolved by moving it entierly. It's worth noting that any software that links with the errno symbol is buggy as errno is defined as a macro that calls a function as permitted by POSIX. I'm not convinced we should be exposing it for linkage at all. -- Brooks
Re: sanitizers broken (was RE: libc/libsys split coming soon) [errno in libsys: any analogy to __elf_aux_vector ?]
Brooks Davis wrote on Date: Thu, 22 Feb 2024 02:03:09 UTC : > On Thu, Feb 22, 2024 at 02:57:13AM +0200, Konstantin Belousov wrote: > > On Wed, Feb 21, 2024 at 08:20:25PM +, Brooks Davis wrote: > > > That's probably worth a shot. Static linking will work anyway because > > > libc.a in effect embeds libsys to retain compatability. > > Please do not add libsys.so to the ABI. Right now it is an implementation > > detail of libthr and libc, and it is preferable to not change it, at least > > not yet, and definitely not to solve some minor internal issues. > > Indeed, on further reflection I agree. Another option occured to me > which I intend to persue tomorrow: explicitly link the sanitizer .so > files with libsys. At least in the base system that should be straight > foward. Does the errno move to libsys have any problems similar to the __elf_aux_vector move to libsys --that might also lead to needing -lsys (for things as the are now)? For reference: https://lists.freebsd.org/archives/dev-commits-src-main/2024-February/022025.html === Mark Millard marklmi at yahoo.com
Re: sanitizers broken (was RE: libc/libsys split coming soon)
Am 2024-02-21 10:52, schrieb hartmut.bra...@dlr.de: Hi, I updated yesterday and now event a minimal program with cc -fsanitize=address produces ld: error: undefined symbol: __elf_aux_vector referenced by sanitizer_linux_libcdep.cpp:950 (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a cc: error: linker command failed with exit code 1 (use -v to see invocation) I think this is caused by the libsys split. There are other issues too. Discussed in multiple places. I opened https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=277222 this morning, maybe it can be used to centralize the libsys issues (= I don't mind of you add a comment there, but maybe brooks wants to have a separate PR). Bye, Alexander. -- http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.orgnetch...@freebsd.org : PGP 0x8F31830F9F2772BF signature.asc Description: OpenPGP digital signature
Re: sanitizers broken (was RE: libc/libsys split coming soon)
On 21 Feb 2024, at 20:00, Brooks Davis wrote: > > The sanitizers reach somewhat questionably into libc internals that are > exported to allow rtld to update them. I was unable to find an solution > that didn't break this and I felt that fixing things like closefrom() > using non-deprecated syscalls was more important than avoiding changes > to the sanitizer interface. On Darwin, Apple added a special __interpose section that contains pairs of functions to be replaced and replacements. Within the library supplying the interposer, the symbol is resolved to the next version along, but everything that links to the interposing library sees the wrapped version. I wonder if it’s worth teaching rtld to do something equivalent. It’s a fairly lightweight generic mechanism that avoids a lot of the hacks that the sanitisers (and other things, such as instrumented malloc wrappers) do. David
Re: sanitizers broken (was RE: libc/libsys split coming soon)
On Thu, Feb 22, 2024 at 02:57:13AM +0200, Konstantin Belousov wrote: > On Wed, Feb 21, 2024 at 08:20:25PM +, Brooks Davis wrote: > > That's probably worth a shot. Static linking will work anyway because > > libc.a in effect embeds libsys to retain compatability. > Please do not add libsys.so to the ABI. Right now it is an implementation > detail of libthr and libc, and it is preferable to not change it, at least > not yet, and definitely not to solve some minor internal issues. Indeed, on further reflection I agree. Another option occured to me which I intend to persue tomorrow: explicitly link the sanitizer .so files with libsys. At least in the base system that should be straight foward. -- Brooks > > > > > -- Brooks > > > > On Wed, Feb 21, 2024 at 09:12:41PM +0100, Dimitry Andric wrote: > > > Can't we just add libsys.so to the /usr/lib/libc.so linker script? That > > > would work for everything except static linking? > > > > > > -Dimitry > > > > > > > On 21 Feb 2024, at 21:00, Brooks Davis wrote: > > > > > > > > TL;DR: you can work around this by adding -lsys to the link line and I > > > > aim to improve the situation soon. > > > > > > > > The sanitizers reach somewhat questionably into libc internals that are > > > > exported to allow rtld to update them. I was unable to find an solution > > > > that didn't break this and I felt that fixing things like closefrom() > > > > using non-deprecated syscalls was more important than avoiding changes > > > > to the sanitizer interface. > > > > > > > > I'm trying to find a way to better solution to the sanitizer. A few > > > > ideas I'm considering: > > > > - Teach clang to add -lsys when linking with sanitizers on sufficently > > > > new systems (con: doesn't fix gcc). > > > > - Make the symbol weak in the sanitizer and complain when it's not > > > > found or call back to using environ. The latter migth have > > > > limitations around direct exec with rtld. > > > > - Relocate __elf_aux_vector to csu so the symbol is always available. > > > > - Adding a new interface to access __elf_aux_vector directly. > > > > > > > > I'll continue to work on this. > > > > > > > > -- Brooks > > > > > > > > On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: > > > >> Hi, > > > >> > > > >> I updated yesterday and now event a minimal program with > > > >> > > > >> cc -fsanitize=address > > > >> > > > >> produces > > > >> > > > >> ld: error: undefined symbol: __elf_aux_vector > > > >>>>> referenced by sanitizer_linux_libcdep.cpp:950 > > > >>>>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) > > > >>>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in > > > >>>>> archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a > > > >> cc: error: linker command failed with exit code 1 (use -v to see > > > >> invocation) > > > >> > > > >> I think this is caused by the libsys split. > > > >> > > > >> Cheers, > > > >> Harti > > > >> > > > >> -Original Message- > > > >> From: owner-freebsd-curr...@freebsd.org > > > >> On Behalf Of Brooks Davis > > > >> Sent: Friday, February 2, 2024 11:32 PM > > > >> To: curr...@freebsd.org > > > >> Subject: libc/libsys split coming soon > > > >> > > > >> TL;DR: The implementation of system calls is moving to a seperate > > > >> library (libsys). No changes are required to existing software > > > >> (except to ensure that libsys is present when building custom disk > > > >> images). > > > >> > > > >> Code: https://github.com/freebsd/freebsd-src/pull/908 > > > >> > > > >> After nearly a decade of intermittent work, I'm about to land a series > > > >> of patches which moves system calls, vdso support, and libc's parsing > > > >> of the ELF auxiliary argument vector into a separate library (libsys). > > > >> I plan to do this early next week (February 5th). > > > >> > > &g
Re: sanitizers broken (was RE: libc/libsys split coming soon)
On Wed, Feb 21, 2024 at 08:20:25PM +, Brooks Davis wrote: > That's probably worth a shot. Static linking will work anyway because > libc.a in effect embeds libsys to retain compatability. Please do not add libsys.so to the ABI. Right now it is an implementation detail of libthr and libc, and it is preferable to not change it, at least not yet, and definitely not to solve some minor internal issues. > > -- Brooks > > On Wed, Feb 21, 2024 at 09:12:41PM +0100, Dimitry Andric wrote: > > Can't we just add libsys.so to the /usr/lib/libc.so linker script? That > > would work for everything except static linking? > > > > -Dimitry > > > > > On 21 Feb 2024, at 21:00, Brooks Davis wrote: > > > > > > TL;DR: you can work around this by adding -lsys to the link line and I > > > aim to improve the situation soon. > > > > > > The sanitizers reach somewhat questionably into libc internals that are > > > exported to allow rtld to update them. I was unable to find an solution > > > that didn't break this and I felt that fixing things like closefrom() > > > using non-deprecated syscalls was more important than avoiding changes > > > to the sanitizer interface. > > > > > > I'm trying to find a way to better solution to the sanitizer. A few > > > ideas I'm considering: > > > - Teach clang to add -lsys when linking with sanitizers on sufficently > > > new systems (con: doesn't fix gcc). > > > - Make the symbol weak in the sanitizer and complain when it's not > > > found or call back to using environ. The latter migth have > > > limitations around direct exec with rtld. > > > - Relocate __elf_aux_vector to csu so the symbol is always available. > > > - Adding a new interface to access __elf_aux_vector directly. > > > > > > I'll continue to work on this. > > > > > > -- Brooks > > > > > > On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: > > >> Hi, > > >> > > >> I updated yesterday and now event a minimal program with > > >> > > >> cc -fsanitize=address > > >> > > >> produces > > >> > > >> ld: error: undefined symbol: __elf_aux_vector > > >>>>> referenced by sanitizer_linux_libcdep.cpp:950 > > >>>>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) > > >>>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in > > >>>>> archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a > > >> cc: error: linker command failed with exit code 1 (use -v to see > > >> invocation) > > >> > > >> I think this is caused by the libsys split. > > >> > > >> Cheers, > > >> Harti > > >> > > >> -Original Message- > > >> From: owner-freebsd-curr...@freebsd.org > > >> On Behalf Of Brooks Davis > > >> Sent: Friday, February 2, 2024 11:32 PM > > >> To: curr...@freebsd.org > > >> Subject: libc/libsys split coming soon > > >> > > >> TL;DR: The implementation of system calls is moving to a seperate > > >> library (libsys). No changes are required to existing software (except > > >> to ensure that libsys is present when building custom disk images). > > >> > > >> Code: https://github.com/freebsd/freebsd-src/pull/908 > > >> > > >> After nearly a decade of intermittent work, I'm about to land a series > > >> of patches which moves system calls, vdso support, and libc's parsing of > > >> the ELF auxiliary argument vector into a separate library (libsys). I > > >> plan to do this early next week (February 5th). > > >> > > >> This change serves three primary purposes: > > >> 1. It's easier to completely replace system call implementations for > > >> tracing or compartmentalization purposes. > > >> 2. It simplifies the implementation of restrictions on system calls such > > >> as those implemented by OpenBSD's msyscall(2) > > >> (https://man.openbsd.org/msyscall.2). > > >> 3. It allows language runtimes to link with libsys for system call > > >> implementations without requiring libc. > > >> > > >> libsys is an auxiliary filter for libc. This means that for any symbol > > >> defined
Re: sanitizers broken (was RE: libc/libsys split coming soon)
That's probably worth a shot. Static linking will work anyway because libc.a in effect embeds libsys to retain compatability. -- Brooks On Wed, Feb 21, 2024 at 09:12:41PM +0100, Dimitry Andric wrote: > Can't we just add libsys.so to the /usr/lib/libc.so linker script? That would > work for everything except static linking? > > -Dimitry > > > On 21 Feb 2024, at 21:00, Brooks Davis wrote: > > > > TL;DR: you can work around this by adding -lsys to the link line and I > > aim to improve the situation soon. > > > > The sanitizers reach somewhat questionably into libc internals that are > > exported to allow rtld to update them. I was unable to find an solution > > that didn't break this and I felt that fixing things like closefrom() > > using non-deprecated syscalls was more important than avoiding changes > > to the sanitizer interface. > > > > I'm trying to find a way to better solution to the sanitizer. A few > > ideas I'm considering: > > - Teach clang to add -lsys when linking with sanitizers on sufficently > > new systems (con: doesn't fix gcc). > > - Make the symbol weak in the sanitizer and complain when it's not > > found or call back to using environ. The latter migth have > > limitations around direct exec with rtld. > > - Relocate __elf_aux_vector to csu so the symbol is always available. > > - Adding a new interface to access __elf_aux_vector directly. > > > > I'll continue to work on this. > > > > -- Brooks > > > > On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: > >> Hi, > >> > >> I updated yesterday and now event a minimal program with > >> > >> cc -fsanitize=address > >> > >> produces > >> > >> ld: error: undefined symbol: __elf_aux_vector > >>>>> referenced by sanitizer_linux_libcdep.cpp:950 > >>>>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) > >>>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in > >>>>> archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a > >> cc: error: linker command failed with exit code 1 (use -v to see > >> invocation) > >> > >> I think this is caused by the libsys split. > >> > >> Cheers, > >> Harti > >> > >> -Original Message- > >> From: owner-freebsd-curr...@freebsd.org > >> On Behalf Of Brooks Davis > >> Sent: Friday, February 2, 2024 11:32 PM > >> To: curr...@freebsd.org > >> Subject: libc/libsys split coming soon > >> > >> TL;DR: The implementation of system calls is moving to a seperate library > >> (libsys). No changes are required to existing software (except to ensure > >> that libsys is present when building custom disk images). > >> > >> Code: https://github.com/freebsd/freebsd-src/pull/908 > >> > >> After nearly a decade of intermittent work, I'm about to land a series of > >> patches which moves system calls, vdso support, and libc's parsing of the > >> ELF auxiliary argument vector into a separate library (libsys). I plan to > >> do this early next week (February 5th). > >> > >> This change serves three primary purposes: > >> 1. It's easier to completely replace system call implementations for > >> tracing or compartmentalization purposes. > >> 2. It simplifies the implementation of restrictions on system calls such > >> as those implemented by OpenBSD's msyscall(2) > >> (https://man.openbsd.org/msyscall.2). > >> 3. It allows language runtimes to link with libsys for system call > >> implementations without requiring libc. > >> > >> libsys is an auxiliary filter for libc. This means that for any symbol > >> defined by both, the libsys version takes precedence at runtime. For > >> system call implementations, libc contains empty stubs. For others it > >> contains copies of the functions (this could be further refined at a later > >> date). The statically linked libc contains the full implementations so > >> linking libsys is not required. > >> > >> Additionally, libthr is now linked with libsys to provide _umtx_op_err(). > >> > >> The overall implementation follows https://reviews.freebsd.org/D14609, > >> but is redone from scratch as multiple commits to facilitate review and > >> assist git's rename detection. > >>
Re: sanitizers broken (was RE: libc/libsys split coming soon)
Can't we just add libsys.so to the /usr/lib/libc.so linker script? That would work for everything except static linking? -Dimitry > On 21 Feb 2024, at 21:00, Brooks Davis wrote: > > TL;DR: you can work around this by adding -lsys to the link line and I > aim to improve the situation soon. > > The sanitizers reach somewhat questionably into libc internals that are > exported to allow rtld to update them. I was unable to find an solution > that didn't break this and I felt that fixing things like closefrom() > using non-deprecated syscalls was more important than avoiding changes > to the sanitizer interface. > > I'm trying to find a way to better solution to the sanitizer. A few > ideas I'm considering: > - Teach clang to add -lsys when linking with sanitizers on sufficently > new systems (con: doesn't fix gcc). > - Make the symbol weak in the sanitizer and complain when it's not > found or call back to using environ. The latter migth have > limitations around direct exec with rtld. > - Relocate __elf_aux_vector to csu so the symbol is always available. > - Adding a new interface to access __elf_aux_vector directly. > > I'll continue to work on this. > > -- Brooks > > On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: >> Hi, >> >> I updated yesterday and now event a minimal program with >> >> cc -fsanitize=address >> >> produces >> >> ld: error: undefined symbol: __elf_aux_vector >>>>> referenced by sanitizer_linux_libcdep.cpp:950 >>>>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) >>>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive >>>>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a >> cc: error: linker command failed with exit code 1 (use -v to see invocation) >> >> I think this is caused by the libsys split. >> >> Cheers, >> Harti >> >> -Original Message- >> From: owner-freebsd-curr...@freebsd.org >> On Behalf Of Brooks Davis >> Sent: Friday, February 2, 2024 11:32 PM >> To: curr...@freebsd.org >> Subject: libc/libsys split coming soon >> >> TL;DR: The implementation of system calls is moving to a seperate library >> (libsys). No changes are required to existing software (except to ensure >> that libsys is present when building custom disk images). >> >> Code: https://github.com/freebsd/freebsd-src/pull/908 >> >> After nearly a decade of intermittent work, I'm about to land a series of >> patches which moves system calls, vdso support, and libc's parsing of the >> ELF auxiliary argument vector into a separate library (libsys). I plan to >> do this early next week (February 5th). >> >> This change serves three primary purposes: >> 1. It's easier to completely replace system call implementations for >> tracing or compartmentalization purposes. >> 2. It simplifies the implementation of restrictions on system calls such >> as those implemented by OpenBSD's msyscall(2) >> (https://man.openbsd.org/msyscall.2). >> 3. It allows language runtimes to link with libsys for system call >> implementations without requiring libc. >> >> libsys is an auxiliary filter for libc. This means that for any symbol >> defined by both, the libsys version takes precedence at runtime. For system >> call implementations, libc contains empty stubs. For others it contains >> copies of the functions (this could be further refined at a later date). >> The statically linked libc contains the full implementations so linking >> libsys is not required. >> >> Additionally, libthr is now linked with libsys to provide _umtx_op_err(). >> >> The overall implementation follows https://reviews.freebsd.org/D14609, >> but is redone from scratch as multiple commits to facilitate review and >> assist git's rename detection. >> >> Testing: >> - Boot testing on amd64, aarch64, and riscv >> - make tinderbox (prior version, final run in progress) >> - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391 >> - Kyua tests in poudriere amd64 jails: same 359 failures as with the >>latest freebsdci build >> >> Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many apologies >> for not landing this in a timely manner. Additional thanks to kib@ for many >> rounds of review, markj@ and kib@ for debugging rtld issues exposed by this >> patch, and antoine@ for exp-runs. >> >> Future work: >> -
Re: sanitizers broken (was RE: libc/libsys split coming soon)
TL;DR: you can work around this by adding -lsys to the link line and I aim to improve the situation soon. The sanitizers reach somewhat questionably into libc internals that are exported to allow rtld to update them. I was unable to find an solution that didn't break this and I felt that fixing things like closefrom() using non-deprecated syscalls was more important than avoiding changes to the sanitizer interface. I'm trying to find a way to better solution to the sanitizer. A few ideas I'm considering: - Teach clang to add -lsys when linking with sanitizers on sufficently new systems (con: doesn't fix gcc). - Make the symbol weak in the sanitizer and complain when it's not found or call back to using environ. The latter migth have limitations around direct exec with rtld. - Relocate __elf_aux_vector to csu so the symbol is always available. - Adding a new interface to access __elf_aux_vector directly. I'll continue to work on this. -- Brooks On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: > Hi, > > I updated yesterday and now event a minimal program with > > cc -fsanitize=address > > produces > > ld: error: undefined symbol: __elf_aux_vector > >>> referenced by sanitizer_linux_libcdep.cpp:950 > >>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) > >>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in > >>> archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a > cc: error: linker command failed with exit code 1 (use -v to see invocation) > > I think this is caused by the libsys split. > > Cheers, > Harti > > -Original Message- > From: owner-freebsd-curr...@freebsd.org > On Behalf Of Brooks Davis > Sent: Friday, February 2, 2024 11:32 PM > To: curr...@freebsd.org > Subject: libc/libsys split coming soon > > TL;DR: The implementation of system calls is moving to a seperate library > (libsys). No changes are required to existing software (except to ensure > that libsys is present when building custom disk images). > > Code: https://github.com/freebsd/freebsd-src/pull/908 > > After nearly a decade of intermittent work, I'm about to land a series of > patches which moves system calls, vdso support, and libc's parsing of the ELF > auxiliary argument vector into a separate library (libsys). I plan to do > this early next week (February 5th). > > This change serves three primary purposes: > 1. It's easier to completely replace system call implementations for > tracing or compartmentalization purposes. > 2. It simplifies the implementation of restrictions on system calls such > as those implemented by OpenBSD's msyscall(2) > (https://man.openbsd.org/msyscall.2). > 3. It allows language runtimes to link with libsys for system call > implementations without requiring libc. > > libsys is an auxiliary filter for libc. This means that for any symbol > defined by both, the libsys version takes precedence at runtime. For system > call implementations, libc contains empty stubs. For others it contains > copies of the functions (this could be further refined at a later date). The > statically linked libc contains the full implementations so linking libsys is > not required. > > Additionally, libthr is now linked with libsys to provide _umtx_op_err(). > > The overall implementation follows https://reviews.freebsd.org/D14609, > but is redone from scratch as multiple commits to facilitate review and > assist git's rename detection. > > Testing: > - Boot testing on amd64, aarch64, and riscv > - make tinderbox (prior version, final run in progress) > - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391 > - Kyua tests in poudriere amd64 jails: same 359 failures as with the > latest freebsdci build > > Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many apologies for > not landing this in a timely manner. Additional thanks to kib@ for many > rounds of review, markj@ and kib@ for debugging rtld issues exposed by this > patch, and antoine@ for exp-runs. > > Future work: > - Purely functional interfaces to system calls (no errorno). > Unfortunately there isn't an obvious way to do this without > significant (possibly generated) assembly code. > - Investigate msyscall(2) and pinsyscalls(2). > - Reduce the size of stubs in libc. Ive errored on the > side of not touching the copies that end up in libc to keep diff > size down. We might want to generate empty stubs instead. > > See also: > - Solaris Linker and Libraries Guide: > https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter4-4.html > > -- Brooks >
Re: sanitizers broken (was RE: libc/libsys split coming soon)
[Brooks' activity related to commit 99ea67573164637d633e8051eb0a5d52f1f9488e
looks likely for what changed the status: "lib{c,sys}: move auxargs more
firmly into libsys".]
On Feb 21, 2024, at 09:02, Mark Millard wrote:
> On Feb 21, 2024, at 08:38, Mark Millard wrote:
>
>> Mark Johnston wrote on
>> Date: Wed, 21 Feb 2024 13:33:43 UTC :
>>
>>> On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote:
Hi,
I updated yesterday and now event a minimal program with
cc -fsanitize=address
produces
ld: error: undefined symbol: __elf_aux_vector
>>> referenced by sanitizer_linux_libcdep.cpp:950
>>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
>>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a
cc: error: linker command failed with exit code 1 (use -v to see
invocation)
I think this is caused by the libsys split.
>>>
>>> I don't see any such problem on a system running 5f7ac491eef4, which
>>> includes the libsys split. Which compiler are you using, and which
>>> revision are you running?
>>
>> Trivial to reproduce via pkgbase install/upgrade:
>>
>> # uname -apKU
>> FreeBSD aarch64-main-pkgs 15.0-CURRENT FreeBSD 15.0-CURRENT
>> main-n268410-445d3d227e68 GENERIC-NODEBUG arm64 aarch64 1500014 1500014
>
> Gack: pkgbase does not have the kernel and world at the same place
> relative to git commits of source code. See my note from yesterday:
>
> https://lists.freebsd.org/archives/freebsd-pkgbase/2024-February/000319.html
>
> Looking at /usr/src/tests/sys/kern/sigsys.c in this pkgbse context
> shows the content from about 10 hours after 445d3d227e68's commit:
>
> git: e53b83a849e3 - main - tests/sigsys: initialize parameter passed to
> sysctlbyname() Gleb Smirnoff
> (CommitDate: 2024-02-20 22:37:45 +)
>
> (Note: The next commit's source was not present.)
>
>
>> # more main.c
>> int main(void) { return 0; }
>>
>> # cc -fsanitize=address main.c
>> ld: error: undefined symbol: __elf_aux_vector
> referenced by sanitizer_linux_libcdep.cpp:950
> (/home/bapt/worktrees/main/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-aarch64.a
> referenced by sanitizer_linux_libcdep.cpp:950
> (/home/bapt/worktrees/main/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-aarch64.a
>> cc: error: linker command failed with exit code 1 (use -v to see invocation)
>
> I do not see the issue in my (somehwat older) personal builds:
>
> # uname -apKU
> you have mail
> FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT #134
> main-n268363-8b67c670a49b-dirty: Sat Feb 17 16:22:31 PST 2024
> root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
> arm64 aarch64 1500014 1500014
>
> (Here kernel and world match git hashes: 8b67c670a49b . But
> my personal builds have patches and other tailoring relative
> to the official git hash they are based on.)
>
My guess is:
Brooks Davis
Date: Mon, 19 Feb 2024 22:44:35 UTC
The branch main has been updated by brooks:
URL:
https://cgit.FreeBSD.org/src/commit/?id=99ea67573164637d633e8051eb0a5d52f1f9488e
commit 99ea67573164637d633e8051eb0a5d52f1f9488e
Author: Brooks Davis
AuthorDate: 2024-02-19 22:44:08 +
Commit: Brooks Davis
CommitDate: 2024-02-19 22:44:08 +
lib{c,sys}: move auxargs more firmly into libsys
Continue to filter the public interface (elf_aux_info()), but entierly
relocate the private interfaces (_elf_aux_info(),
__init_elf_aux_vector(), and __elf_aux_vector) to libsys.
This ensures that rtld updates the correct (only) copy of
__elf_aux_vector. After 968a18975adc9c2a619bb52aa2f009de99fc9e24
updates were confused and __getosreldate was failing, causing
the system to fall back to compat compat12 syscalls in some cases.
Return to explicitly linking libc to libsys and link libthr with libc
and libsys (in that order).
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D43910
. .
===
Mark Millard
marklmi at yahoo.com
Re: sanitizers broken (was RE: libc/libsys split coming soon)
On Feb 21, 2024, at 08:38, Mark Millard wrote:
> Mark Johnston wrote on
> Date: Wed, 21 Feb 2024 13:33:43 UTC :
>
>> On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote:
>>> Hi,
>>>
>>> I updated yesterday and now event a minimal program with
>>>
>>> cc -fsanitize=address
>>>
>>> produces
>>>
>>> ld: error: undefined symbol: __elf_aux_vector
>> referenced by sanitizer_linux_libcdep.cpp:950
>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a
>>> cc: error: linker command failed with exit code 1 (use -v to see invocation)
>>>
>>> I think this is caused by the libsys split.
>>
>> I don't see any such problem on a system running 5f7ac491eef4, which
>> includes the libsys split. Which compiler are you using, and which
>> revision are you running?
>
> Trivial to reproduce via pkgbase install/upgrade:
>
> # uname -apKU
> FreeBSD aarch64-main-pkgs 15.0-CURRENT FreeBSD 15.0-CURRENT
> main-n268410-445d3d227e68 GENERIC-NODEBUG arm64 aarch64 1500014 1500014
Gack: pkgbase does not have the kernel and world at the same place
relative to git commits of source code. See my note from yesterday:
https://lists.freebsd.org/archives/freebsd-pkgbase/2024-February/000319.html
Looking at /usr/src/tests/sys/kern/sigsys.c in this pkgbse context
shows the content from about 10 hours after 445d3d227e68's commit:
git: e53b83a849e3 - main - tests/sigsys: initialize parameter passed to
sysctlbyname() Gleb Smirnoff
(CommitDate: 2024-02-20 22:37:45 +)
(Note: The next commit's source was not present.)
> # more main.c
> int main(void) { return 0; }
>
> # cc -fsanitize=address main.c
> ld: error: undefined symbol: __elf_aux_vector
referenced by sanitizer_linux_libcdep.cpp:950
(/home/bapt/worktrees/main/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
/usr/lib/clang/17/lib/freebsd/libclang_rt.asan-aarch64.a
referenced by sanitizer_linux_libcdep.cpp:950
(/home/bapt/worktrees/main/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
/usr/lib/clang/17/lib/freebsd/libclang_rt.asan-aarch64.a
> cc: error: linker command failed with exit code 1 (use -v to see invocation)
I do not see the issue in my (somehwat older) personal builds:
# uname -apKU
you have mail
FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT #134
main-n268363-8b67c670a49b-dirty: Sat Feb 17 16:22:31 PST 2024
root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
arm64 aarch64 1500014 1500014
(Here kernel and world match git hashes: 8b67c670a49b . But
my personal builds have patches and other tailoring relative
to the official git hash they are based on.)
===
Mark Millard
marklmi at yahoo.com
Re: sanitizers broken (was RE: libc/libsys split coming soon)
Mark Johnston wrote on
Date: Wed, 21 Feb 2024 13:33:43 UTC :
> On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote:
> > Hi,
> >
> > I updated yesterday and now event a minimal program with
> >
> > cc -fsanitize=address
> >
> > produces
> >
> > ld: error: undefined symbol: __elf_aux_vector
> > >>> referenced by sanitizer_linux_libcdep.cpp:950
> > >>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
> > >>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
> > >>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a
> > cc: error: linker command failed with exit code 1 (use -v to see invocation)
> >
> > I think this is caused by the libsys split.
>
> I don't see any such problem on a system running 5f7ac491eef4, which
> includes the libsys split. Which compiler are you using, and which
> revision are you running?
Trivial to reproduce via pkgbase install/upgrade:
# uname -apKU
FreeBSD aarch64-main-pkgs 15.0-CURRENT FreeBSD 15.0-CURRENT
main-n268410-445d3d227e68 GENERIC-NODEBUG arm64 aarch64 1500014 1500014
# more main.c
int main(void) { return 0; }
# cc -fsanitize=address main.c
ld: error: undefined symbol: __elf_aux_vector
>>> referenced by sanitizer_linux_libcdep.cpp:950
>>> (/home/bapt/worktrees/main/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
>>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-aarch64.a
>>> referenced by sanitizer_linux_libcdep.cpp:950
>>> (/home/bapt/worktrees/main/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950)
>>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive
>>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-aarch64.a
cc: error: linker command failed with exit code 1 (use -v to see invocation)
===
Mark Millard
marklmi at yahoo.com
Re: sanitizers broken (was RE: libc/libsys split coming soon)
On Wed, 21 Feb 2024, Mark Johnston wrote: MJ>On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: MJ>> Hi, MJ>> MJ>> I updated yesterday and now event a minimal program with MJ>> MJ>> cc -fsanitize=address MJ>> MJ>> produces MJ>> MJ>> ld: error: undefined symbol: __elf_aux_vector MJ>> >>> referenced by sanitizer_linux_libcdep.cpp:950 (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) MJ>> >>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a MJ>> cc: error: linker command failed with exit code 1 (use -v to see invocation) MJ>> MJ>> I think this is caused by the libsys split. MJ> MJ>I don't see any such problem on a system running 5f7ac491eef4, which MJ>includes the libsys split. Which compiler are you using, and which MJ>revision are you running? That is 445d3d227e68. The compiler is the system cc. Harti MJ> MJ>> Cheers, MJ>> Harti MJ>> MJ>> -Original Message- MJ>> From: owner-freebsd-curr...@freebsd.org On Behalf Of Brooks Davis MJ>> Sent: Friday, February 2, 2024 11:32 PM MJ>> To: curr...@freebsd.org MJ>> Subject: libc/libsys split coming soon MJ>> MJ>> TL;DR: The implementation of system calls is moving to a seperate library (libsys). No changes are required to existing software (except to ensure that libsys is present when building custom disk images). MJ>> MJ>> Code: https://github.com/freebsd/freebsd-src/pull/908 MJ>> MJ>> After nearly a decade of intermittent work, I'm about to land a series of patches which moves system calls, vdso support, and libc's parsing of the ELF auxiliary argument vector into a separate library (libsys). I plan to do this early next week (February 5th). MJ>> MJ>> This change serves three primary purposes: MJ>> 1. It's easier to completely replace system call implementations for MJ>> tracing or compartmentalization purposes. MJ>> 2. It simplifies the implementation of restrictions on system calls such MJ>> as those implemented by OpenBSD's msyscall(2) MJ>> (https://man.openbsd.org/msyscall.2). MJ>> 3. It allows language runtimes to link with libsys for system call MJ>> implementations without requiring libc. MJ>> MJ>> libsys is an auxiliary filter for libc. This means that for any symbol defined by both, the libsys version takes precedence at runtime. For system call implementations, libc contains empty stubs. For others it contains copies of the functions (this could be further refined at a later date). The statically linked libc contains the full implementations so linking libsys is not required. MJ>> MJ>> Additionally, libthr is now linked with libsys to provide _umtx_op_err(). MJ>> MJ>> The overall implementation follows https://reviews.freebsd.org/D14609, MJ>> but is redone from scratch as multiple commits to facilitate review and assist git's rename detection. MJ>> MJ>> Testing: MJ>> - Boot testing on amd64, aarch64, and riscv MJ>> - make tinderbox (prior version, final run in progress) MJ>> - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391 MJ>> - Kyua tests in poudriere amd64 jails: same 359 failures as with the MJ>> latest freebsdci build MJ>> MJ>> Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many apologies for not landing this in a timely manner. Additional thanks to kib@ for many rounds of review, markj@ and kib@ for debugging rtld issues exposed by this patch, and antoine@ for exp-runs. MJ>> MJ>> Future work: MJ>> - Purely functional interfaces to system calls (no errorno). MJ>> Unfortunately there isn't an obvious way to do this without MJ>> significant (possibly generated) assembly code. MJ>> - Investigate msyscall(2) and pinsyscalls(2). MJ>> - Reduce the size of stubs in libc. I’ve errored on the MJ>> side of not touching the copies that end up in libc to keep diff MJ>> size down. We might want to generate empty stubs instead. MJ>> MJ>> See also: MJ>> - Solaris Linker and Libraries Guide: MJ>> https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter4-4.html MJ>> MJ>> -- Brooks MJ>> MJ>
Re: sanitizers broken (was RE: libc/libsys split coming soon)
On Wed, Feb 21, 2024 at 09:52:23AM +, hartmut.bra...@dlr.de wrote: > Hi, > > I updated yesterday and now event a minimal program with > > cc -fsanitize=address > > produces > > ld: error: undefined symbol: __elf_aux_vector > >>> referenced by sanitizer_linux_libcdep.cpp:950 > >>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) > >>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in > >>> archive /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a > cc: error: linker command failed with exit code 1 (use -v to see invocation) > > I think this is caused by the libsys split. I don't see any such problem on a system running 5f7ac491eef4, which includes the libsys split. Which compiler are you using, and which revision are you running? > Cheers, > Harti > > -Original Message- > From: owner-freebsd-curr...@freebsd.org > On Behalf Of Brooks Davis > Sent: Friday, February 2, 2024 11:32 PM > To: curr...@freebsd.org > Subject: libc/libsys split coming soon > > TL;DR: The implementation of system calls is moving to a seperate library > (libsys). No changes are required to existing software (except to ensure > that libsys is present when building custom disk images). > > Code: https://github.com/freebsd/freebsd-src/pull/908 > > After nearly a decade of intermittent work, I'm about to land a series of > patches which moves system calls, vdso support, and libc's parsing of the ELF > auxiliary argument vector into a separate library (libsys). I plan to do > this early next week (February 5th). > > This change serves three primary purposes: > 1. It's easier to completely replace system call implementations for > tracing or compartmentalization purposes. > 2. It simplifies the implementation of restrictions on system calls such > as those implemented by OpenBSD's msyscall(2) > (https://man.openbsd.org/msyscall.2). > 3. It allows language runtimes to link with libsys for system call > implementations without requiring libc. > > libsys is an auxiliary filter for libc. This means that for any symbol > defined by both, the libsys version takes precedence at runtime. For system > call implementations, libc contains empty stubs. For others it contains > copies of the functions (this could be further refined at a later date). The > statically linked libc contains the full implementations so linking libsys is > not required. > > Additionally, libthr is now linked with libsys to provide _umtx_op_err(). > > The overall implementation follows https://reviews.freebsd.org/D14609, > but is redone from scratch as multiple commits to facilitate review and > assist git's rename detection. > > Testing: > - Boot testing on amd64, aarch64, and riscv > - make tinderbox (prior version, final run in progress) > - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391 > - Kyua tests in poudriere amd64 jails: same 359 failures as with the > latest freebsdci build > > Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many apologies for > not landing this in a timely manner. Additional thanks to kib@ for many > rounds of review, markj@ and kib@ for debugging rtld issues exposed by this > patch, and antoine@ for exp-runs. > > Future work: > - Purely functional interfaces to system calls (no errorno). > Unfortunately there isn't an obvious way to do this without > significant (possibly generated) assembly code. > - Investigate msyscall(2) and pinsyscalls(2). > - Reduce the size of stubs in libc. I’ve errored on the > side of not touching the copies that end up in libc to keep diff > size down. We might want to generate empty stubs instead. > > See also: > - Solaris Linker and Libraries Guide: > https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter4-4.html > > -- Brooks >
sanitizers broken (was RE: libc/libsys split coming soon)
Hi, I updated yesterday and now event a minimal program with cc -fsanitize=address produces ld: error: undefined symbol: __elf_aux_vector >>> referenced by sanitizer_linux_libcdep.cpp:950 >>> (/usr/src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:950) >>> sanitizer_linux_libcdep.o:(__sanitizer::ReExec()) in archive >>> /usr/lib/clang/17/lib/freebsd/libclang_rt.asan-x86_64.a cc: error: linker command failed with exit code 1 (use -v to see invocation) I think this is caused by the libsys split. Cheers, Harti -Original Message- From: owner-freebsd-curr...@freebsd.org On Behalf Of Brooks Davis Sent: Friday, February 2, 2024 11:32 PM To: curr...@freebsd.org Subject: libc/libsys split coming soon TL;DR: The implementation of system calls is moving to a seperate library (libsys). No changes are required to existing software (except to ensure that libsys is present when building custom disk images). Code: https://github.com/freebsd/freebsd-src/pull/908 After nearly a decade of intermittent work, I'm about to land a series of patches which moves system calls, vdso support, and libc's parsing of the ELF auxiliary argument vector into a separate library (libsys). I plan to do this early next week (February 5th). This change serves three primary purposes: 1. It's easier to completely replace system call implementations for tracing or compartmentalization purposes. 2. It simplifies the implementation of restrictions on system calls such as those implemented by OpenBSD's msyscall(2) (https://man.openbsd.org/msyscall.2). 3. It allows language runtimes to link with libsys for system call implementations without requiring libc. libsys is an auxiliary filter for libc. This means that for any symbol defined by both, the libsys version takes precedence at runtime. For system call implementations, libc contains empty stubs. For others it contains copies of the functions (this could be further refined at a later date). The statically linked libc contains the full implementations so linking libsys is not required. Additionally, libthr is now linked with libsys to provide _umtx_op_err(). The overall implementation follows https://reviews.freebsd.org/D14609, but is redone from scratch as multiple commits to facilitate review and assist git's rename detection. Testing: - Boot testing on amd64, aarch64, and riscv - make tinderbox (prior version, final run in progress) - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391 - Kyua tests in poudriere amd64 jails: same 359 failures as with the latest freebsdci build Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many apologies for not landing this in a timely manner. Additional thanks to kib@ for many rounds of review, markj@ and kib@ for debugging rtld issues exposed by this patch, and antoine@ for exp-runs. Future work: - Purely functional interfaces to system calls (no errorno). Unfortunately there isn't an obvious way to do this without significant (possibly generated) assembly code. - Investigate msyscall(2) and pinsyscalls(2). - Reduce the size of stubs in libc. I’ve errored on the side of not touching the copies that end up in libc to keep diff size down. We might want to generate empty stubs instead. See also: - Solaris Linker and Libraries Guide: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter4-4.html -- Brooks
Re: libc/libsys split coming soon
Hi > On 5 Feb 2024, at 17:02, Brooks Davis wrote: > > >>> 2. It simplifies the implementation of restrictions on system calls such >>> as those implemented by OpenBSD's msyscall(2) >>> (https://man.openbsd.org/msyscall.2). >> >> That's one to ignore for tools that make syscalls outside of the libc memory >> mapping. > > Should someone implement msyscall(2) there will certainly be an opt out > mechanism along the usual lines that uses elfctl and procctl. I was thinking more of simply not passing on such syscalls. > >>> 3. It allows language runtimes to link with libsys for system call >>> implementations without requiring libc. >> >> I see that pagesize is on the list of functions that are moving. There are a >> couple of other functions that might cause me problems if libc isn't linked. >> >> Could you do a quick test with an exe linked to libsys but not libc running >> under Valgrind memcheck, please? > > Could you suggest a more concrete example? I don’t think that it really matters. Just an empty main(). The important thing would be linking to libsys and not libc. A+ Paul
Re: libc/libsys split coming soon
On Sat, Feb 03, 2024 at 10:15:09AM +0100, Mateusz Guzik wrote: > On 2/2/24, Brooks Davis wrote: > > TL;DR: The implementation of system calls is moving to a seperate > > library (libsys). No changes are required to existing software (except > > to ensure that libsys is present when building custom disk images). > > > > Code: https://github.com/freebsd/freebsd-src/pull/908 > > > > After nearly a decade of intermittent work, I'm about to land a series > > of patches which moves system calls, vdso support, and libc's parsing of > > the ELF auxiliary argument vector into a separate library (libsys). I > > plan to do this early next week (February 5th). > > > > This change serves three primary purposes: > > 1. It's easier to completely replace system call implementations for > > tracing or compartmentalization purposes. > > 2. It simplifies the implementation of restrictions on system calls such > > as those implemented by OpenBSD's msyscall(2) > > (https://man.openbsd.org/msyscall.2). > > 3. It allows language runtimes to link with libsys for system call > > implementations without requiring libc. > > > > libsys is an auxiliary filter for libc. This means that for any symbol > > defined by both, the libsys version takes precedence at runtime. For > > system call implementations, libc contains empty stubs. For others it > > contains copies of the functions (this could be further refined at a > > later date). The statically linked libc contains the full > > implementations so linking libsys is not required. > > > > Do I read it correctly that everything dynamically linked will also be > linked to libsys, as in executing such a binary will now require > loading one extra .so? > > Binary startup is very slow, for example execve of a hello world > binary in a Linux-based chroot on FreeBSD is faster by a factor of 2 > compared to a native one. As such perf-wise this looks like a step in > the wrong direction. > > Is there a problem making it so that libc ends up unchanged, but all > the bits are available separately in libsys if one does not want libc? Doing so fails to accomplish goal 1 for people who want to do it system wide and does nothing for goal 2. That being said, it would be pretty easy to make this something we can toggle if we find it's an issue in practice. That being said, it seems like we should be spending time on rtld improvements instead of optimizing for its current (seemingly worse than necessary) performance. -- Brooks
Re: libc/libsys split coming soon
On Sat, Feb 03, 2024 at 05:11:42PM +0100, Paul Floyd wrote: > > On 02/02/2024 23:31, Brooks Davis wrote: > > TL;DR: The implementation of system calls is moving to a seperate > > library (libsys). No changes are required to existing software (except > > to ensure that libsys is present when building custom disk images). > > > > Code:https://github.com/freebsd/freebsd-src/pull/908 > > > > After nearly a decade of intermittent work, I'm about to land a series > > of patches which moves system calls, vdso support, and libc's parsing of > > the ELF auxiliary argument vector into a separate library (libsys). I > > plan to do this early next week (February 5th). > > > > This change serves three primary purposes: > >1. It's easier to completely replace system call implementations for > > tracing or compartmentalization purposes. > > Will that affect code that makes syscalls without currently linking to libc? Such programs might be broken in these environments but it would depend a lot on the details of both the program and the replacement syscall layer. > >2. It simplifies the implementation of restrictions on system calls such > > as those implemented by OpenBSD's msyscall(2) > > (https://man.openbsd.org/msyscall.2). > > That's one to ignore for tools that make syscalls outside of the libc memory > mapping. Should someone implement msyscall(2) there will certainly be an opt out mechanism along the usual lines that uses elfctl and procctl. > >3. It allows language runtimes to link with libsys for system call > > implementations without requiring libc. > > I see that pagesize is on the list of functions that are moving. There are a > couple of other functions that might cause me problems if libc isn't linked. > > Could you do a quick test with an exe linked to libsys but not libc running > under Valgrind memcheck, please? Could you suggest a more concrete example? -- Brooks
Re: libc/libsys split coming soon
On Sat, Feb 03, 2024 at 11:10:14AM -0700, Warner Losh wrote: > On Sat, Feb 3, 2024, 11:02 AM Konstantin Belousov > wrote: > > > On Sat, Feb 03, 2024 at 11:05:10AM -0500, Daniel Eischen wrote: > > > Will this break binary compatibility with older programs expecting those > > symbols in libc and not linked to libsys? > > > > As was mentioned, libc filters libsys. This means that libc exports all > > the same symbols as before, but forward the implementation to libsys. > > For apps nothing changes, the introduction of libsys is (should be) ABI > > compatible. > > > > More, I would state that no binaries wanting to state binary-compatble > > with future FreeBSD should link to libsys directly, at least for now. > > > > How do you view Golang or Rust run times using this then? They try to avoid > libc today. Goland runtime issues syscalls directly (using CPU instructions). I believe this is true even when Go-compiled binary is linked against libc to provide C FFI. Rust does use libc to get system services. No changes there. Or, is you question about switching Go and Rust to directly using libsys? Then Rust does not need that. For Go, indeed, using either libc or libsys instead of static linking and directly issuing syscalls would be better. Right now Go binary needs to understand all small details in the difference between C wrappers ABI and real syscalls. Also they re-wrote e.g. the gettime() code from vdso into their runtime. > > Warner > > Warner > > > > > > > > On Feb 3, 2024, at 3:39 AM, Dave Cottlehuber > > wrote: > > > > > > > > On Fri, 2 Feb 2024, at 23:31, Brooks Davis wrote: > > > >> TL;DR: The implementation of system calls is moving to a seperate > > > >> library (libsys). No changes are required to existing software > > (except > > > >> to ensure that libsys is present when building custom disk images). > > > >> > > > >> Code: https://github.com/freebsd/freebsd-src/pull/908 > > > >> > > > >> After nearly a decade of intermittent work, I'm about to land a series > > > >> of patches which moves system calls, vdso support, and libc's parsing > > of > > > >> the ELF auxiliary argument vector into a separate library (libsys). I > > > >> plan to do this early next week (February 5th). > > > >> > > > >> This change serves three primary purposes: > > > >> 1. It's easier to completely replace system call implementations for > > > >> tracing or compartmentalization purposes. > > > >> 2. It simplifies the implementation of restrictions on system calls > > such > > > >> as those implemented by OpenBSD's msyscall(2) > > > >> (https://man.openbsd.org/msyscall.2). > > > >> 3. It allows language runtimes to link with libsys for system call > > > >> implementations without requiring libc. > > > > > > > > Awesome! So (3) is generally considered ideal for languages like > > zig[1], rust or go, to use directly? > > > > > > > > What’s the appropriate mechanism for such a language to know which > > version of FreeBSD it’s talking to, to ensure syscall table matches the > > languages expectations? > > > > > > > > It would be nice to hear about any experiments in (2) and how that > > compares to things such as capsicum. > > > > > > > > [1]: https://github.com/ziglang/zig/issues/165 > > > > > > > > A+ > > > > Dave > > > > > > > > > > > > > > > > > >
Re: libc/libsys split coming soon
On Sat, Feb 3, 2024, 11:02 AM Konstantin Belousov wrote: > On Sat, Feb 03, 2024 at 11:05:10AM -0500, Daniel Eischen wrote: > > Will this break binary compatibility with older programs expecting those > symbols in libc and not linked to libsys? > > As was mentioned, libc filters libsys. This means that libc exports all > the same symbols as before, but forward the implementation to libsys. > For apps nothing changes, the introduction of libsys is (should be) ABI > compatible. > > More, I would state that no binaries wanting to state binary-compatble > with future FreeBSD should link to libsys directly, at least for now. > How do you view Golang or Rust run times using this then? They try to avoid libc today. Warner Warner > > > > > On Feb 3, 2024, at 3:39 AM, Dave Cottlehuber > wrote: > > > > > > On Fri, 2 Feb 2024, at 23:31, Brooks Davis wrote: > > >> TL;DR: The implementation of system calls is moving to a seperate > > >> library (libsys). No changes are required to existing software > (except > > >> to ensure that libsys is present when building custom disk images). > > >> > > >> Code: https://github.com/freebsd/freebsd-src/pull/908 > > >> > > >> After nearly a decade of intermittent work, I'm about to land a series > > >> of patches which moves system calls, vdso support, and libc's parsing > of > > >> the ELF auxiliary argument vector into a separate library (libsys). I > > >> plan to do this early next week (February 5th). > > >> > > >> This change serves three primary purposes: > > >> 1. It's easier to completely replace system call implementations for > > >> tracing or compartmentalization purposes. > > >> 2. It simplifies the implementation of restrictions on system calls > such > > >> as those implemented by OpenBSD's msyscall(2) > > >> (https://man.openbsd.org/msyscall.2). > > >> 3. It allows language runtimes to link with libsys for system call > > >> implementations without requiring libc. > > > > > > Awesome! So (3) is generally considered ideal for languages like > zig[1], rust or go, to use directly? > > > > > > What’s the appropriate mechanism for such a language to know which > version of FreeBSD it’s talking to, to ensure syscall table matches the > languages expectations? > > > > > > It would be nice to hear about any experiments in (2) and how that > compares to things such as capsicum. > > > > > > [1]: https://github.com/ziglang/zig/issues/165 > > > > > > A+ > > > Dave > > > > > > > > > > > >
Re: libc/libsys split coming soon
On Sat, Feb 03, 2024 at 11:05:10AM -0500, Daniel Eischen wrote: > Will this break binary compatibility with older programs expecting those > symbols in libc and not linked to libsys? As was mentioned, libc filters libsys. This means that libc exports all the same symbols as before, but forward the implementation to libsys. For apps nothing changes, the introduction of libsys is (should be) ABI compatible. More, I would state that no binaries wanting to state binary-compatble with future FreeBSD should link to libsys directly, at least for now. > > > On Feb 3, 2024, at 3:39 AM, Dave Cottlehuber wrote: > > > > On Fri, 2 Feb 2024, at 23:31, Brooks Davis wrote: > >> TL;DR: The implementation of system calls is moving to a seperate > >> library (libsys). No changes are required to existing software (except > >> to ensure that libsys is present when building custom disk images). > >> > >> Code: https://github.com/freebsd/freebsd-src/pull/908 > >> > >> After nearly a decade of intermittent work, I'm about to land a series > >> of patches which moves system calls, vdso support, and libc's parsing of > >> the ELF auxiliary argument vector into a separate library (libsys). I > >> plan to do this early next week (February 5th). > >> > >> This change serves three primary purposes: > >> 1. It's easier to completely replace system call implementations for > >> tracing or compartmentalization purposes. > >> 2. It simplifies the implementation of restrictions on system calls such > >> as those implemented by OpenBSD's msyscall(2) > >> (https://man.openbsd.org/msyscall.2). > >> 3. It allows language runtimes to link with libsys for system call > >> implementations without requiring libc. > > > > Awesome! So (3) is generally considered ideal for languages like zig[1], > > rust or go, to use directly? > > > > What’s the appropriate mechanism for such a language to know which version > > of FreeBSD it’s talking to, to ensure syscall table matches the languages > > expectations? > > > > It would be nice to hear about any experiments in (2) and how that compares > > to things such as capsicum. > > > > [1]: https://github.com/ziglang/zig/issues/165 > > > > A+ > > Dave > > > > > >
Re: libc/libsys split coming soon
On Sat, Feb 03, 2024 at 12:12:35PM +0100, Mateusz Guzik wrote: > On 2/3/24, David Chisnall wrote: > > On 3 Feb 2024, at 09:15, Mateusz Guzik wrote: > >> > >> Binary startup is very slow, for example execve of a hello world > >> binary in a Linux-based chroot on FreeBSD is faster by a factor of 2 > >> compared to a native one. As such perf-wise this looks like a step in > >> the wrong direction. It is the right change to improve modularity and the structure of the code. > > > > Have you profiled this? Is the Linux version using BIND_NOW (which comes > > with a load of problems, but it often the default for Linux systems and > > reduces the number of slow-path entries into rtld)? Do they trigger the > > same number of CoW faults? Is there a path in rtld that’s slower than the > > equivalent ld-linux.so path? Linux version probably benefits from pre-linking, which might have the side-effect of breaking semantic into as if BIND_NOW is activated. > > > > I only profiled FreeBSD, it was 4 years ago. I have neither time nor > interest in working on this. > > Relevant excerpts from profiling an fexecve loop: > > Sampling what syscalls was being executed when in kernel mode > (or trap): > > syscalls: >pread 108 >fstat 162 >issetugid 250 > sigprocmask 303 > read 310 > mprotect 341 > open 380 >close 1547 > mmap 2787 > trap 5421 > [snip] > In userspace most of the time is spent here: > ld-elf.so.1`memset 406 > ld-elf.so.1`matched_symbol 431 > ld-elf.so.1`strcmp 1078 >ld-elf.so.1`reloc_non_plt 1102 > ld-elf.so.1`symlook_obj 1102 > ld-elf.so.1`find_symdef 1439 > > find_symdef iterates a linked list, which I presume induces strcmp calls > due to unwanted entries. > [snip] So strcmp() is almost 1:1 with reloc_non_plt and/or symlook_obj. It demonstrates that the ELF hash (perhaps GNU hash, but I do not remember how long do we have it) provides very good distribution.
Re: libc/libsys split coming soon
On 02/02/2024 23:31, Brooks Davis wrote: TL;DR: The implementation of system calls is moving to a seperate library (libsys). No changes are required to existing software (except to ensure that libsys is present when building custom disk images). Code:https://github.com/freebsd/freebsd-src/pull/908 After nearly a decade of intermittent work, I'm about to land a series of patches which moves system calls, vdso support, and libc's parsing of the ELF auxiliary argument vector into a separate library (libsys). I plan to do this early next week (February 5th). This change serves three primary purposes: 1. It's easier to completely replace system call implementations for tracing or compartmentalization purposes. Will that affect code that makes syscalls without currently linking to libc? 2. It simplifies the implementation of restrictions on system calls such as those implemented by OpenBSD's msyscall(2) (https://man.openbsd.org/msyscall.2). That's one to ignore for tools that make syscalls outside of the libc memory mapping. 3. It allows language runtimes to link with libsys for system call implementations without requiring libc. I see that pagesize is on the list of functions that are moving. There are a couple of other functions that might cause me problems if libc isn't linked. Could you do a quick test with an exe linked to libsys but not libc running under Valgrind memcheck, please? A+ Paul
Re: libc/libsys split coming soon
Will this break binary compatibility with older programs expecting those symbols in libc and not linked to libsys? > On Feb 3, 2024, at 3:39 AM, Dave Cottlehuber wrote: > > On Fri, 2 Feb 2024, at 23:31, Brooks Davis wrote: >> TL;DR: The implementation of system calls is moving to a seperate >> library (libsys). No changes are required to existing software (except >> to ensure that libsys is present when building custom disk images). >> >> Code: https://github.com/freebsd/freebsd-src/pull/908 >> >> After nearly a decade of intermittent work, I'm about to land a series >> of patches which moves system calls, vdso support, and libc's parsing of >> the ELF auxiliary argument vector into a separate library (libsys). I >> plan to do this early next week (February 5th). >> >> This change serves three primary purposes: >> 1. It's easier to completely replace system call implementations for >> tracing or compartmentalization purposes. >> 2. It simplifies the implementation of restrictions on system calls such >> as those implemented by OpenBSD's msyscall(2) >> (https://man.openbsd.org/msyscall.2). >> 3. It allows language runtimes to link with libsys for system call >> implementations without requiring libc. > > Awesome! So (3) is generally considered ideal for languages like zig[1], rust > or go, to use directly? > > What’s the appropriate mechanism for such a language to know which version of > FreeBSD it’s talking to, to ensure syscall table matches the languages > expectations? > > It would be nice to hear about any experiments in (2) and how that compares > to things such as capsicum. > > [1]: https://github.com/ziglang/zig/issues/165 > > A+ > Dave > >
Re: libc/libsys split coming soon
Am Sat, Feb 03, 2024 at 10:15:09AM +0100 schrieb Mateusz Guzik: > Do I read it correctly that everything dynamically linked will also be > linked to libsys, as in executing such a binary will now require > loading one extra .so? > > Binary startup is very slow, for example execve of a hello world > binary in a Linux-based chroot on FreeBSD is faster by a factor of 2 > compared to a native one. As such perf-wise this looks like a step in > the wrong direction. I wonder if we could follow the steps of musl libc and just integrate libsys/libc into rtld, as basically all dynamically linked programs link these libraries anyway. Yours, Robert Clausecker -- () ascii ribbon campaign - for an encoding-agnostic world /\ - against html email - against proprietary attachments
Re: libc/libsys split coming soon
On 2/3/24, David Chisnall wrote:
> On 3 Feb 2024, at 09:15, Mateusz Guzik wrote:
>>
>> Binary startup is very slow, for example execve of a hello world
>> binary in a Linux-based chroot on FreeBSD is faster by a factor of 2
>> compared to a native one. As such perf-wise this looks like a step in
>> the wrong direction.
>
> Have you profiled this? Is the Linux version using BIND_NOW (which comes
> with a load of problems, but it often the default for Linux systems and
> reduces the number of slow-path entries into rtld)? Do they trigger the
> same number of CoW faults? Is there a path in rtld that’s slower than the
> equivalent ld-linux.so path?
>
I only profiled FreeBSD, it was 4 years ago. I have neither time nor
interest in working on this.
Relevant excerpts from profiling an fexecve loop:
Sampling what syscalls was being executed when in kernel mode
(or trap):
syscalls:
pread 108
fstat 162
issetugid 250
sigprocmask 303
read 310
mprotect 341
open 380
close 1547
mmap 2787
trap 5421
[snip]
In userspace most of the time is spent here:
ld-elf.so.1`memset 406
ld-elf.so.1`matched_symbol 431
ld-elf.so.1`strcmp 1078
ld-elf.so.1`reloc_non_plt 1102
ld-elf.so.1`symlook_obj 1102
ld-elf.so.1`find_symdef 1439
find_symdef iterates a linked list, which I presume induces strcmp calls
due to unwanted entries.
[snip]
Full profile
user:
libc.so.7`__je_extent_heap_new 71
libc.so.7`__vdso_clock_gettime 73
libc.so.7`memset 75
ld-elf.so.1`_rtld 83
ld-elf.so.1`getenv 85
libc.so.7`__je_malloc_mutex_boot 132
ld-elf.so.1`reloc_plt 148
ld-elf.so.1`__crt_malloc 163
ld-elf.so.1`symlook_default 166
ld-elf.so.1`digest_dynamic1 184
libc.so.7`__je_malloc_mutex_init 205
ld-elf.so.1`symlook_global 281
ld-elf.so.1`memset 406
ld-elf.so.1`matched_symbol 431
ld-elf.so.1`strcmp 1078
ld-elf.so.1`reloc_non_plt 1102
ld-elf.so.1`symlook_obj 1102
ld-elf.so.1`find_symdef 1439
kernel:
kernel`vm_reserv_alloc_page 89
kernel`amd64_syscall 95
kernel`0x80 102
kernel`vm_page_alloc_domain_after 114
kernel`vm_object_deallocate 117
kernel`vm_map_pmap_enter 122
kernel`pmap_enter_object 140
kernel`uma_zalloc_arg 148
kernel`vm_map_lookup_entry 148
kernel`pmap_try_insert_pv_entry 156
kernel`vm_fault_dirty 168
kernel`pagecopy 177
kernel`vm_fault 260
kernel`get_pv_entry 265
kernel`pagezero_erms 367
kernel`pmap_enter_quick_locked 380
kernel`pmap_enter 432
kernel`0x80 1126
kernel`0x80 2017
kernel`trap 2097
syscalls:
pread 108
fstat 162
issetugid 250
sigprocmask 303
read 310
mprotect 341
open 380
close 1547
mmap 2787
trap 5421
Counting fexecve:
dtrace -n 'fbt::sys_fexecve:entry { @[count] = stack(); } tick-30s { exit(0); }'
dtrace script, can be run as: dtrace -w -x aggsize=128M -s script.d
assumes binary name is a.out
syscall::fexecve:entry
{
self->inexec = 1;
}
syscall::fexecve:return
{
self->inexec = 0;
}
fbt::trap:entry
{
self->trap = 1;
}
fbt::trap:return
{
self->trap = 0;
}
Re: libc/libsys split coming soon
On 3 Feb 2024, at 09:15, Mateusz Guzik wrote: > > Binary startup is very slow, for example execve of a hello world > binary in a Linux-based chroot on FreeBSD is faster by a factor of 2 > compared to a native one. As such perf-wise this looks like a step in > the wrong direction. Have you profiled this? Is the Linux version using BIND_NOW (which comes with a load of problems, but it often the default for Linux systems and reduces the number of slow-path entries into rtld)? Do they trigger the same number of CoW faults? Is there a path in rtld that’s slower than the equivalent ld-linux.so path? David
Re: libc/libsys split coming soon
On 2/2/24, Brooks Davis wrote: > TL;DR: The implementation of system calls is moving to a seperate > library (libsys). No changes are required to existing software (except > to ensure that libsys is present when building custom disk images). > > Code: https://github.com/freebsd/freebsd-src/pull/908 > > After nearly a decade of intermittent work, I'm about to land a series > of patches which moves system calls, vdso support, and libc's parsing of > the ELF auxiliary argument vector into a separate library (libsys). I > plan to do this early next week (February 5th). > > This change serves three primary purposes: > 1. It's easier to completely replace system call implementations for > tracing or compartmentalization purposes. > 2. It simplifies the implementation of restrictions on system calls such > as those implemented by OpenBSD's msyscall(2) > (https://man.openbsd.org/msyscall.2). > 3. It allows language runtimes to link with libsys for system call > implementations without requiring libc. > > libsys is an auxiliary filter for libc. This means that for any symbol > defined by both, the libsys version takes precedence at runtime. For > system call implementations, libc contains empty stubs. For others it > contains copies of the functions (this could be further refined at a > later date). The statically linked libc contains the full > implementations so linking libsys is not required. > Do I read it correctly that everything dynamically linked will also be linked to libsys, as in executing such a binary will now require loading one extra .so? Binary startup is very slow, for example execve of a hello world binary in a Linux-based chroot on FreeBSD is faster by a factor of 2 compared to a native one. As such perf-wise this looks like a step in the wrong direction. Is there a problem making it so that libc ends up unchanged, but all the bits are available separately in libsys if one does not want libc? -- Mateusz Guzik
Re: libc/libsys split coming soon
On Fri, 2 Feb 2024, at 23:31, Brooks Davis wrote: > TL;DR: The implementation of system calls is moving to a seperate > library (libsys). No changes are required to existing software (except > to ensure that libsys is present when building custom disk images). > > Code: https://github.com/freebsd/freebsd-src/pull/908 > > After nearly a decade of intermittent work, I'm about to land a series > of patches which moves system calls, vdso support, and libc's parsing of > the ELF auxiliary argument vector into a separate library (libsys). I > plan to do this early next week (February 5th). > > This change serves three primary purposes: > 1. It's easier to completely replace system call implementations for > tracing or compartmentalization purposes. > 2. It simplifies the implementation of restrictions on system calls such > as those implemented by OpenBSD's msyscall(2) > (https://man.openbsd.org/msyscall.2). > 3. It allows language runtimes to link with libsys for system call > implementations without requiring libc. Awesome! So (3) is generally considered ideal for languages like zig[1], rust or go, to use directly? What’s the appropriate mechanism for such a language to know which version of FreeBSD it’s talking to, to ensure syscall table matches the languages expectations? It would be nice to hear about any experiments in (2) and how that compares to things such as capsicum. [1]: https://github.com/ziglang/zig/issues/165 A+ Dave
Re: libc/libsys split coming soon
Brooks Davis wrote in : |TL;DR: The implementation of system calls is moving to a seperate |library (libsys). No changes are required to existing software (except |to ensure that libsys is present when building custom disk images). ... [] |This change serves three primary purposes: | 1. It's easier to completely replace system call implementations for | tracing or compartmentalization purposes. | 2. It simplifies the implementation of restrictions on system calls such | as those implemented by OpenBSD's msyscall(2) | (https://man.openbsd.org/msyscall.2). | 3. It allows language runtimes to link with libsys for system call | implementations without requiring libc. That is so cool. Much love for 3.! .. |After nearly a decade of intermittent work, I'm about to land a series |of patches which moves system calls, vdso support, and libc's parsing of |the ELF auxiliary argument vector into a separate library (libsys). I |plan to do this early next week (February 5th). Congratulations. Thanks for all your efforts. All FreeBSD! Oh i had a mail in the queue for another list, and whereas i admire all efforts, how i love this one! Thank you, and nice weekend i wish from Germany! --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt)
libc/libsys split coming soon
TL;DR: The implementation of system calls is moving to a seperate library (libsys). No changes are required to existing software (except to ensure that libsys is present when building custom disk images). Code: https://github.com/freebsd/freebsd-src/pull/908 After nearly a decade of intermittent work, I'm about to land a series of patches which moves system calls, vdso support, and libc's parsing of the ELF auxiliary argument vector into a separate library (libsys). I plan to do this early next week (February 5th). This change serves three primary purposes: 1. It's easier to completely replace system call implementations for tracing or compartmentalization purposes. 2. It simplifies the implementation of restrictions on system calls such as those implemented by OpenBSD's msyscall(2) (https://man.openbsd.org/msyscall.2). 3. It allows language runtimes to link with libsys for system call implementations without requiring libc. libsys is an auxiliary filter for libc. This means that for any symbol defined by both, the libsys version takes precedence at runtime. For system call implementations, libc contains empty stubs. For others it contains copies of the functions (this could be further refined at a later date). The statically linked libc contains the full implementations so linking libsys is not required. Additionally, libthr is now linked with libsys to provide _umtx_op_err(). The overall implementation follows https://reviews.freebsd.org/D14609, but is redone from scratch as multiple commits to facilitate review and assist git's rename detection. Testing: - Boot testing on amd64, aarch64, and riscv - make tinderbox (prior version, final run in progress) - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391 - Kyua tests in poudriere amd64 jails: same 359 failures as with the latest freebsdci build Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many apologies for not landing this in a timely manner. Additional thanks to kib@ for many rounds of review, markj@ and kib@ for debugging rtld issues exposed by this patch, and antoine@ for exp-runs. Future work: - Purely functional interfaces to system calls (no errorno). Unfortunately there isn't an obvious way to do this without significant (possibly generated) assembly code. - Investigate msyscall(2) and pinsyscalls(2). - Reduce the size of stubs in libc. I???ve errored on the side of not touching the copies that end up in libc to keep diff size down. We might want to generate empty stubs instead. See also: - Solaris Linker and Libraries Guide: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter4-4.html -- Brooks
