On 2/3/24, David Chisnall <thera...@freebsd.org> wrote:
> On 3 Feb 2024, at 09:15, Mateusz Guzik <mjgu...@gmail.com> wrote:
>>
>> Binary startup is very slow, for example execve of a hello world
>> binary in a Linux-based chroot on FreeBSD is faster by a factor of 2
>> compared to a native one. As such perf-wise this looks like a step in
>> the wrong direction.
>
> Have you profiled this? Is the Linux version using BIND_NOW (which comes
> with a load of problems, but it often the default for Linux systems and
> reduces the number of slow-path entries into rtld)? Do they trigger the
> same number of CoW faults? Is there a path in rtld that’s slower than the
> equivalent ld-linux.so path?
>
I only profiled FreeBSD, it was 4 years ago. I have neither time nor
interest in working on this.
Relevant excerpts from profiling an fexecve loop:
Sampling what syscalls was being executed when in kernel mode
(or trap):
syscalls:
pread 108
fstat 162
issetugid 250
sigprocmask 303
read 310
mprotect 341
open 380
close 1547
mmap 2787
trap 5421
[snip]
In userspace most of the time is spent here:
ld-elf.so.1`memset 406
ld-elf.so.1`matched_symbol 431
ld-elf.so.1`strcmp 1078
ld-elf.so.1`reloc_non_plt 1102
ld-elf.so.1`symlook_obj 1102
ld-elf.so.1`find_symdef 1439
find_symdef iterates a linked list, which I presume induces strcmp calls
due to unwanted entries.
[snip]
Full profile
user:
libc.so.7`__je_extent_heap_new 71
libc.so.7`__vdso_clock_gettime 73
libc.so.7`memset 75
ld-elf.so.1`_rtld 83
ld-elf.so.1`getenv 85
libc.so.7`__je_malloc_mutex_boot 132
ld-elf.so.1`reloc_plt 148
ld-elf.so.1`__crt_malloc 163
ld-elf.so.1`symlook_default 166
ld-elf.so.1`digest_dynamic1 184
libc.so.7`__je_malloc_mutex_init 205
ld-elf.so.1`symlook_global 281
ld-elf.so.1`memset 406
ld-elf.so.1`matched_symbol 431
ld-elf.so.1`strcmp 1078
ld-elf.so.1`reloc_non_plt 1102
ld-elf.so.1`symlook_obj 1102
ld-elf.so.1`find_symdef 1439
kernel:
kernel`vm_reserv_alloc_page 89
kernel`amd64_syscall 95
kernel`0xffffffff80 102
kernel`vm_page_alloc_domain_after 114
kernel`vm_object_deallocate 117
kernel`vm_map_pmap_enter 122
kernel`pmap_enter_object 140
kernel`uma_zalloc_arg 148
kernel`vm_map_lookup_entry 148
kernel`pmap_try_insert_pv_entry 156
kernel`vm_fault_dirty 168
kernel`pagecopy 177
kernel`vm_fault 260
kernel`get_pv_entry 265
kernel`pagezero_erms 367
kernel`pmap_enter_quick_locked 380
kernel`pmap_enter 432
kernel`0xffffffff80 1126
kernel`0xffffffff80 2017
kernel`trap 2097
syscalls:
pread 108
fstat 162
issetugid 250
sigprocmask 303
read 310
mprotect 341
open 380
close 1547
mmap 2787
trap 5421
Counting fexecve:
dtrace -n 'fbt::sys_fexecve:entry { @[count] = stack(); } tick-30s { exit(0); }'
dtrace script, can be run as: dtrace -w -x aggsize=128M -s script.d
assumes binary name is a.out
syscall::fexecve:entry
{
self->inexec = 1;
}
syscall::fexecve:return
{
self->inexec = 0;
}
fbt::trap:entry
{
self->trap = 1;
}
fbt::trap:return
{
self->trap = 0;
}
profile:::profile-4999
/execname == "a.out" && arg1 && self->inexec == 0/
{
@user[usym(arg1)] = count();
}
profile:::profile-4999
/execname == "a.out" && arg0 && self->inexec == 0 && self->trap == 0/
{
@kernel[sym(arg0)] = count();
@kernel_syscall[stringof(`syscallnames[curthread->td_sa.code])]
= count();
}
profile:::profile-4999
/execname == "a.out" && arg0 && self->inexec == 0 && self->trap == 1/
{
@kernel[sym(arg0)] = count();
@kernel_syscall["trap"] = count();
}
tick-5s
{
system("clear");
trunc(@user, 20);
trunc(@kernel, 20);
printa("%40A %@16d\n", @user);
printa("%40a %@16d\n", @kernel);
clear(@user);
clear(@kernel);
trunc(@kernel_syscall, 10);
printf("\t\t\tsyscalls:\n");
printa("%40s %@16d\n", @kernel_syscall);
clear(@kernel_syscall);
}
fexecve-loop.c:
#include <sys/types.h>
#include <sys/time.h>
#include <err.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
int
main(int argc, char **argv)
{
char buf[8];
char *cargv[3];
int fd;
switch (argc) {
case 1:
fd = open(argv[0], O_EXEC);
if (fd == -1)
err(1, "open");
cargv[0] = argv[0];
cargv[1] = buf;
sprintf(cargv[1], "%d", fd);
cargv[2] = NULL;
break;
case 2:
fd = atoi(argv[1]);
cargv[0] = argv[0];
cargv[1] = argv[1];
cargv[2] = NULL;
break;
default:
printf("invalid argc %d\n", argc);
exit(1);
}
fexecve(fd, cargv, NULL);
err(1, "fexecve");
--
Mateusz Guzik <mjguzik gmail.com>
