Thanks for putting so much effort into figuring that out! Really appreciate 
it, and glad to get it working!

On Wednesday, December 9, 2020 at 3:45:30 PM UTC-7 Matthew Kenigsberg wrote:

> That worked!!! Had to set -z relro -z lazy
>
> On Wednesday, December 9, 2020 at 12:30:55 PM UTC-7 [email protected] 
> wrote:
>
>> Hi,
>>
>> Thanks for uploading the files. It definitely has helped me figure out 
>> the issue. 
>>
>> In essence, all the .so files like libzfs.so are built with Full RELRO 
>> (run 'readelf -a libzfso.so | grep BIND_NOW) on Nix OS it looks like. 
>> Relatedly, some Linux distributions are setup to make gcc effectively use 
>> '-z now -z relro' when linking the libraries. On many others like Ubuntu or 
>> Fedora they are built with Partial RELRO - '-z relro' by default.
>>
>> As the libraries are loaded by OSv dynamic linker, all jump slot 
>> relocations are resolved eagerly (even if they are not used by the code 
>> later) if those libraries are marked as 'Full RELRO' (bind_now = true). For 
>> non-'Full RELRO' cases, the jump slot relocations are resolved lazily 
>> whenever they are accessed 1st time and are handled by 'void* 
>> object::resolve_pltgot(unsigned index)` which writes resolved function 
>> symbol address in GOT.
>>
>> The problem with Full-RELRO is that if we cannot find a symbol because 
>> for example it is not implemented by OSv or is not visible at *this 
>> point of linking* we simply ignore it hoping that it will never be used 
>> or resolved later. If it is used later, the resolve_pltgot() is called, and 
>> if the symbol is found (because the library containing the symbol has been 
>> loaded since) we crash because we trying to write to the part of memory - 
>> GOT - that has been since read-only protected.
>>
>> Why does this happen exactly?
>>
>> So here is the symbol *bsd_getmntany *not found at the address you were 
>> getting original fault at (after adding extra debug statements):
>> ELF [tid:28, mod:3, /libzfs.so]: arch_relocate_jump_slot, 
>> addr:0x100000040ca0
>> /libzfs.so: ignoring missing symbol *bsd_getmntany //Would have been 
>> *0x100000040ca8 
>> which match what page fault reports
>> ELF [tid:28, mod:3, /libzfs.so]: arch_relocate_jump_slot, 
>> addr:0x100000040cb0
>>
>> Please note that both mkfs.so and zfs.so depend on libzfs.so. Per command 
>> line, OSv loads and executes the apps sequentially. So the mkfs.so is first 
>> and the dynamic linker will load libzfs.so and relocate and eagerly resolve 
>> all symbols and fix the permissions on libzfs.so. One of the symbols in 
>> libzfs.so is *bsd_getmntany* which is actually part of zfs.so which is 
>> left unresolved (see missing warning above).
>>
>> After mkfs.so, OSv gets to zfs.so and it processes it and executes and 
>> some of the code in zfs.so tries to invoke *bsd_getmntany *which gets 
>> dynamically resolved and found by resolve_pltgot() BUT when it tries to 
>> write to GOT it gets page fault.
>>
>> Having said all that I am sure what exactly the problem is:
>> A) Invalid or abnormal dependency between libzfs.so, mkfs.so and zfs.so 
>> which effectively prevents those to function properly if build with Full 
>> RELRO (would that work on Linux)?
>> B) True limitation of OSv linker which should handle correctly such 
>> scenario.
>>
>> For now, the easiest solution (might be true if A is true) is to simply 
>> force building those libraries with Partial RELRO like in this patch:
>>
>> diff --git a/Makefile b/Makefile
>> index d1597263..d200dde8 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -345,7 +345,7 @@ $(out)/%.o: %.s
>>         $(makedir)
>>         $(call quiet, $(CXX) $(CXXFLAGS) $(ASFLAGS) -c -o $@ $<, AS $*.s)
>>  
>> -%.so: EXTRA_FLAGS = -fPIC -shared
>> +%.so: EXTRA_FLAGS = -fPIC -shared -z relro
>>  %.so: %.o
>>         $(makedir)
>>         $(q-build-so)
>>
>> Please let me know if it works,
>> Waldek
>>
>> PS. Also verify that running ' readelf -a libzfso.so | grep BIND_NOW' 
>> does not show anything anymore.
>>
>> On Tuesday, December 8, 2020 at 5:08:18 PM UTC-5 Matthew Kenigsberg wrote:
>>
>>> Not completely sure where libgcc_s.so.1 is coming from, but I uploaded what 
>>> I have in 
>>> /nix/store/vran8acwir59772hj4vscr7zribvp7l5-gcc-9.3.0-lib/lib/libgcc_s.so.1:
>>> https://drive.google.com/drive/folders/1rM6g-FrzwFpuHr2wX9-J21DzSjyQXGg2
>>>
>>> Get a different error if I comment out core/elf.cc:1429:
>>>
>>> (gdb) bt
>>> #0 0x000000004039eef2 in processor::cli_hlt () at 
>>> arch/x64/processor.hh:247
>>> #1 arch::halt_no_interrupts () at arch/x64/arch.hh:48
>>> #2 osv::halt () at arch/x64/power.cc:26
>>> #3 0x000000004023c73f in abort (fmt=fmt@entry=0x40645aff "Aborted\n") at 
>>> runtime.cc:132
>>> #4 0x0000000040202989 in abort () at runtime.cc:98
>>> #5 0x0000000040218943 in osv::generate_signal (siginfo=..., 
>>> ef=0xffff80000191c068) at libc/signal.cc:124
>>> #6 0x00000000404745ff in osv::handle_mmap_fault (addr=<optimized out>, 
>>> sig=<optimized out>, ef=<optimized out>)
>>> at libc/signal.cc:139
>>> #7 0x0000000040347872 in mmu::vm_fault (addr=17592187039744, 
>>> addr@entry=17592187040376,
>>> ef=ef@entry=0xffff80000191c068) at core/mmu.cc:1336
>>> #8 0x00000000403992e3 in page_fault (ef=0xffff80000191c068) at 
>>> arch/x64/mmu.cc:42
>>> #9 <signal handler called>
>>> #10 0x00001000001554c4 in usage (requested=requested@entry=false)
>>> at bsd/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:424
>>> #11 0x0000100000152025 in main (argc=5, argv=0xffffa00000f19400)
>>> at bsd/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:6676
>>> #12 0x000000004043c311 in osv::application::run_main 
>>> (this=0xffffa00000ee0c10)
>>> at 
>>> /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/stl_vector.h:915
>>> #13 0x000000004022452f in osv::application::main 
>>> (this=0xffffa00000ee0c10) at core/app.cc:320
>>> #14 0x000000004043c539 in osv::application::<lambda(void*)>::operator() 
>>> (__closure=0x0, app=<optimized out>)
>>> at core/app.cc:233
>>> #15 osv::application::<lambda(void*)>::_FUN(void *) () at core/app.cc:235
>>> #16 0x0000000040470d58 in 
>>> pthread_private::pthread::<lambda()>::operator() 
>>> (__closure=0xffffa00001067f00)
>>> at libc/pthread.cc:115
>>> #17 std::_Function_handler<void(), 
>>> pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const 
>>> pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const 
>>> std::_Any_data &) (__functor=...)
>>> at 
>>> /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300
>>> #18 0x00000000404074fd in sched::thread_main_c (t=0xffff800001917040) at 
>>> arch/x64/arch-switch.hh:325
>>> #19 0x00000000403990d3 in thread_main () at arch/x64/entry.S:113
>>>
>>>
>>> On Tuesday, December 8, 2020 at 2:34:19 PM UTC-7 [email protected] 
>>> wrote:
>>>
>>>> I wonder if we have a bug in core/elf.cc::fix_permissions() or logic 
>>>> around. And we might be making the wrong part of the mapping readable 
>>>> based 
>>>> on GNU_RELRO header. I wonder if you are able to create ZFS image by 
>>>> temporarily commenting out the line 1429 of core/elf.cc:
>>>>
>>>> ef->fix_permissions();
>>>> Also, would it possible to get copies of those binaries:
>>>> /libenviron.so: libenviron.so
>>>> /libvdso.so: libvdso.so
>>>> /zpool.so: zpool.so
>>>> /libzfs.so: libzfs.so
>>>> /libuutil.so: libuutil.so
>>>> /zfs.so: zfs.so
>>>> /tools/mkfs.so: tools/mkfs/mkfs.so
>>>> /tools/cpiod.so: tools/cpiod/cpiod.so
>>>> /tools/mount-fs.so: tools/mount/mount-fs.so
>>>> /tools/umount.so: tools/mount/umount.so
>>>> /usr/lib/libgcc_s.so.1: %(libgcc_s_dir)s/libgcc_s.so.1
>>>>
>>>> Ideally, the stripped versions. That would help me to re-create the 
>>>> problem and investigate further.
>>>>
>>>> On Tuesday, December 8, 2020 at 1:25:10 PM UTC-5 Matthew Kenigsberg 
>>>> wrote:
>>>>
>>>>> [nix-shell:~/osv]$ readelf -l build/release/libzfs-stripped.so
>>>>>
>>>>> Elf file type is DYN (Shared object file)
>>>>> Entry point 0xc8f0
>>>>> There are 8 program headers, starting at offset 64
>>>>>
>>>>> Program Headers:
>>>>> Type Offset VirtAddr PhysAddr
>>>>> FileSiz MemSiz Flags Align
>>>>> LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
>>>>> 0x000000000000a9b8 0x000000000000a9b8 R 0x1000
>>>>> LOAD 0x000000000000b000 0x000000000000b000 0x000000000000b000
>>>>> 0x000000000001e0a9 0x000000000001e0a9 R E 0x1000
>>>>> LOAD 0x000000000002a000 0x000000000002a000 0x000000000002a000
>>>>> 0x00000000000093a0 0x00000000000093a0 R 0x1000
>>>>> LOAD 0x0000000000034010 0x0000000000035010 0x0000000000035010
>>>>> 0x0000000000001810 0x0000000000002c20 RW 0x1000
>>>>> DYNAMIC 0x00000000000340e0 0x00000000000350e0 0x00000000000350e0
>>>>> 0x0000000000000210 0x0000000000000210 RW 0x8
>>>>> GNU_EH_FRAME 0x000000000002e768 0x000000000002e768 0x000000000002e768
>>>>> 0x0000000000000d04 0x0000000000000d04 R 0x4
>>>>> GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
>>>>> 0x0000000000000000 0x0000000000000000 RW 0x10
>>>>> GNU_RELRO 0x0000000000034010 0x0000000000035010 0x0000000000035010
>>>>> 0x0000000000000ff0 0x0000000000000ff0 R 0x1
>>>>>
>>>>> Section to Segment mapping:
>>>>> Segment Sections...
>>>>> 00 .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r 
>>>>> .rela.dyn .rela.plt 
>>>>> 01 .init .plt .plt.got .text .fini 
>>>>> 02 .rodata .eh_frame_hdr .eh_frame 
>>>>> 03 .init_array .fini_array .data.rel.ro .dynamic .got .data .bss 
>>>>> 04 .dynamic 
>>>>> 05 .eh_frame_hdr 
>>>>> 06 
>>>>> 07 .init_array .fini_array .data.rel.ro .dynamic .got
>>>>>
>>>>> On Tuesday, December 8, 2020 at 11:17:46 AM UTC-7 [email protected] 
>>>>> wrote:
>>>>>
>>>>>> Back to why it is failing.
>>>>>>
>>>>>> Based on what you sent us:
>>>>>> ..
>>>>>> 0x000010000000b000 0x0000100000016000 [44.0 kB] flags=fmF perm=r 
>>>>>> offset=0x00000000 path=/libzfs.so
>>>>>> 0x0000100000016000 0x0000100000035000 [124.0 kB] flags=fmF perm=rx 
>>>>>> offset=0x0000b000 path=/libzfs.so
>>>>>> 0x0000100000035000 0x000010000003f000 [40.0 kB] flags=fmF perm=r 
>>>>>> offset=0x0002a000 path=/libzfs.so
>>>>>>
>>>>>> *0x0000100000040000 0x0000100000041000 [4.0 kB] flags=fmF perm=r 
>>>>>> offset=0x00034000 path=/libzfs.so*0x0000100000041000 
>>>>>> 0x0000100000042000 [4.0 kB] flags=fmF perm=rw offset=0x00035000 
>>>>>> path=/libzfs.so
>>>>>> ..
>>>>>>
>>>>>> The page fault in arch_relocate_jump_slot() is caused by an attempt 
>>>>>> to write at the address 0x100000040ca8 which falls into a 4th mapping 
>>>>>> range 
>>>>>> from the above which can only be read from. So that is the permission 
>>>>>> fault. The question is why the address is in that range? The address 
>>>>>> should 
>>>>>> be somewhere in GOT in the 5th range - 0x0000100000041000 
>>>>>> 0x0000100000042000 [4.0 kB] flags=fmF perm=rw offset=0x00035000 
>>>>>> path=/libzfs.so which has read/write permission.
>>>>>>
>>>>>> On Ubuntu host when I run the same command like and add extra 
>>>>>> debugging to print statements like this:
>>>>>>
>>>>>> ELF [tid:36, mod:5, /*libzfs*.so]: arch_relocate_jump_slot, 
>>>>>> addr:0x10000007a8d0
>>>>>> They all print addresses within the range 0x10000007a000 - 
>>>>>> 0x10000007b000 which are read-write permitted as they should be:
>>>>>> 0x0000100000044000 0x000010000004e000 [40.0 kB]        flags=fmF      
>>>>>> perm=r    offset=0x00000000 path=/libzfs.so
>>>>>> 0x000010000004e000 0x000010000006f000 [132.0 kB]       flags=fmF      
>>>>>> perm=rx   offset=0x0000a000 path=/libzfs.so
>>>>>> 0x000010000006f000 0x0000100000079000 [40.0 kB]        flags=fmF      
>>>>>> perm=r    offset=0x0002b000 path=/libzfs.so
>>>>>> 0x0000100000079000 0x000010000007a000 [4.0 kB]         flags=fmF      
>>>>>> perm=r    offset=0x00034000 path=/libzfs.so
>>>>>> *0x000010000007a000 0x000010000007c000 [8.0 kB]         flags=fmF    
>>>>>>   perm=rw   offset=0x00035000 path=/libzfs.so *
>>>>>>  
>>>>>> I wonder if we have a bug when calculating where each segment should 
>>>>>> be mapped:
>>>>>>
>>>>>> 400 void file::load_segment(const Elf64_Phdr& phdr)
>>>>>>  401 {
>>>>>>  402     ulong vstart = align_down(phdr.p_vaddr, mmu::page_size);
>>>>>>  403     ulong filesz_unaligned = phdr.p_vaddr + phdr.p_filesz - 
>>>>>> vstart;
>>>>>>  404     ulong filesz = align_up(filesz_unaligned, mmu::page_size);
>>>>>>  405     ulong memsz = align_up(phdr.p_vaddr + phdr.p_memsz, 
>>>>>> mmu::page_size) - vstart;
>>>>>>  406 
>>>>>>  407     unsigned perm = get_segment_mmap_permissions(phdr);
>>>>>>  408 
>>>>>>  409     auto flag = mmu::mmap_fixed | (mlocked() ? 
>>>>>> mmu::mmap_populate : 0);
>>>>>>  410     mmu::map_file(_base + vstart, filesz, flag, perm, _f, 
>>>>>> align_down(phdr.p_offset, mmu::page_size));
>>>>>>  411     if (phdr.p_filesz != phdr.p_memsz) {
>>>>>>  412         assert(perm & mmu::perm_write);
>>>>>>  413         memset(_base + vstart + filesz_unaligned, 0, filesz - 
>>>>>> filesz_unaligned);
>>>>>>  414         if (memsz != filesz) {
>>>>>>  415             mmu::map_anon(_base + vstart + filesz, memsz - 
>>>>>> filesz, flag, perm);
>>>>>>  416         }
>>>>>>  417     }
>>>>>>  418     elf_debug("Loaded and mapped PT_LOAD segment at: %018p of 
>>>>>> size: 0x%x\n", _base + vstart, filesz);
>>>>>>  419 }
>>>>>>
>>>>>> BTW I am also interested what the output of this readelf command for 
>>>>>> your libzfs.so is. Mine is this:
>>>>>>
>>>>>>  readelf -l build/release/libzfs-stripped.so 
>>>>>>
>>>>>> Elf file type is DYN (Shared object file)
>>>>>> Entry point 0xd1d0
>>>>>> There are 11 program headers, starting at offset 64
>>>>>>
>>>>>> Program Headers:
>>>>>>   Type           Offset             VirtAddr           PhysAddr
>>>>>>                  FileSiz            MemSiz              Flags  Align
>>>>>>   LOAD           0x0000000000000000 0x0000000000000000 
>>>>>> 0x0000000000000000
>>>>>>                  0x00000000000098e0 0x00000000000098e0  R      0x1000
>>>>>>   LOAD           0x000000000000a000 0x000000000000a000 
>>>>>> 0x000000000000a000
>>>>>>                  0x00000000000201a1 0x00000000000201a1  R E    0x1000
>>>>>>   LOAD           0x000000000002b000 0x000000000002b000 
>>>>>> 0x000000000002b000
>>>>>>                  0x0000000000009258 0x0000000000009258  R      0x1000
>>>>>>   LOAD           0x0000000000034cb0 0x0000000000035cb0 
>>>>>> 0x0000000000035cb0
>>>>>>                  0x00000000000017f0 0x0000000000002c00  RW     0x1000
>>>>>>   DYNAMIC        0x0000000000034d80 0x0000000000035d80 
>>>>>> 0x0000000000035d80
>>>>>>                  0x00000000000001d0 0x00000000000001d0  RW     0x8
>>>>>>   NOTE           0x00000000000002a8 0x00000000000002a8 
>>>>>> 0x00000000000002a8
>>>>>>                  0x0000000000000020 0x0000000000000020  R      0x8
>>>>>>   NOTE           0x00000000000002c8 0x00000000000002c8 
>>>>>> 0x00000000000002c8
>>>>>>                  0x0000000000000024 0x0000000000000024  R      0x4
>>>>>>   GNU_PROPERTY   0x00000000000002a8 0x00000000000002a8 
>>>>>> 0x00000000000002a8
>>>>>>                  0x0000000000000020 0x0000000000000020  R      0x8
>>>>>>   GNU_EH_FRAME   0x000000000002f768 0x000000000002f768 
>>>>>> 0x000000000002f768
>>>>>>                  0x0000000000000cec 0x0000000000000cec  R      0x4
>>>>>>   GNU_STACK      0x0000000000000000 0x0000000000000000 
>>>>>> 0x0000000000000000
>>>>>>                  0x0000000000000000 0x0000000000000000  RW     0x10
>>>>>>   GNU_RELRO      0x0000000000034cb0 0x0000000000035cb0 
>>>>>> 0x0000000000035cb0
>>>>>>                  0x0000000000000350 0x0000000000000350  R      0x1
>>>>>>
>>>>>>  Section to Segment mapping:
>>>>>>   Segment Sections...
>>>>>>    00     .note.gnu.property .note.gnu.build-id .gnu.hash .dynsym 
>>>>>> .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt 
>>>>>>    01     .init .plt .plt.got .plt.sec .text .fini 
>>>>>>    02     .rodata .eh_frame_hdr .eh_frame 
>>>>>>    03     .init_array .fini_array .data.rel.ro .dynamic .got 
>>>>>> .got.plt .data .bss 
>>>>>>    04     .dynamic 
>>>>>>    05     .note.gnu.property 
>>>>>>    06     .note.gnu.build-id 
>>>>>>    07     .note.gnu.property 
>>>>>>    08     .eh_frame_hdr 
>>>>>>    09     
>>>>>>    10     .init_array .fini_array .data.rel.ro .dynamic .got 
>>>>>>
>>>>>> On Tuesday, December 8, 2020 at 11:39:43 AM UTC-5 Matthew Kenigsberg 
>>>>>> wrote:
>>>>>>
>>>>>>> My gdb is not the strongest but if I hbreak on 
>>>>>>> arch_relocate_jump_slot looks like _pathname is /libzfs.so, eventually 
>>>>>>> /zpool.so, and then a single /libzfs.so before continue hangs
>>>>>>>
>>>>>>> On Tuesday, December 8, 2020 at 9:11:15 AM UTC-7 Matthew Kenigsberg 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Nix is a package manager, and NixOS is an operating system built 
>>>>>>>> completely around the package manager. So every library is stored 
>>>>>>>> somewhere 
>>>>>>>> in /nix/store, like for example on Nix there is never anything like 
>>>>>>>> /lib64/ld-linux-x86-64.so. 
>>>>>>>> It would be /nix/store/.../ld-linux-x86-64.so. I could install the 
>>>>>>>> package 
>>>>>>>> manager on a different OS, in which case I might have both /lib64 and 
>>>>>>>> /nix/store, but on NixOS I'll just have the latter. Does that make 
>>>>>>>> sense? Not 
>>>>>>>> sure if that's messing up something with linking. Guessing I can't 
>>>>>>>> reproduce the error on any other OS, but happy to try.
>>>>>>>> On Tuesday, December 8, 2020 at 9:05:11 AM UTC-7 Matthew Kenigsberg 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> (gdb) connect
>>>>>>>>> abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: 
>>>>>>>>> %d)\n") at runtime.cc:105
>>>>>>>>> 105 do {} while (true);
>>>>>>>>> (gdb) osv syms
>>>>>>>>> manifest.find_file: path=/libvdso.so, found file=libvdso.so
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so 0x100000000000
>>>>>>>>> add symbol table from file 
>>>>>>>>> "/home/matthew/osv/build/release.x64/libvdso.so" at
>>>>>>>>> .text_addr = 0x100000001040
>>>>>>>>> .hash_addr = 0x1000000001c8
>>>>>>>>> .gnu.hash_addr = 0x100000000200
>>>>>>>>> .dynsym_addr = 0x100000000238
>>>>>>>>> .dynstr_addr = 0x1000000002f8
>>>>>>>>> .gnu.version_addr = 0x1000000003be
>>>>>>>>> .gnu.version_d_addr = 0x1000000003d0
>>>>>>>>> .rela.plt_addr = 0x100000000408
>>>>>>>>> .plt_addr = 0x100000001000
>>>>>>>>> .eh_frame_addr = 0x100000002000
>>>>>>>>> .dynamic_addr = 0x100000003e60
>>>>>>>>> .got_addr = 0x100000003fd0
>>>>>>>>> .comment_addr = 0x100000000000
>>>>>>>>> .debug_aranges_addr = 0x100000000000
>>>>>>>>> .debug_info_addr = 0x100000000000
>>>>>>>>> .debug_abbrev_addr = 0x100000000000
>>>>>>>>> .debug_line_addr = 0x100000000000
>>>>>>>>> .debug_str_addr = 0x100000000000
>>>>>>>>> .debug_loc_addr = 0x100000000000
>>>>>>>>> .symtab_addr = 0x100000000000
>>>>>>>>> .strtab_addr = 0x100000000000
>>>>>>>>> warning: section .comment not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .debug_aranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .debug_info not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .debug_abbrev not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .debug_line not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .debug_str not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .debug_loc not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .symtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> warning: section .strtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libvdso.so
>>>>>>>>> manifest.find_file: path=/tools/mkfs.so, found 
>>>>>>>>> file=tools/mkfs/mkfs.so
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so 
>>>>>>>>> 0x100000004000
>>>>>>>>> add symbol table from file 
>>>>>>>>> "/home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so" at
>>>>>>>>> .text_addr = 0x100000006250
>>>>>>>>> .hash_addr = 0x100000004200
>>>>>>>>> .gnu.hash_addr = 0x100000004360
>>>>>>>>> .dynsym_addr = 0x1000000043c0
>>>>>>>>> .dynstr_addr = 0x100000004840
>>>>>>>>> .gnu.version_addr = 0x100000005092
>>>>>>>>> .gnu.version_r_addr = 0x1000000050f8
>>>>>>>>> .rela.dyn_addr = 0x100000005148
>>>>>>>>> .rela.plt_addr = 0x100000005298
>>>>>>>>> .init_addr = 0x100000006000
>>>>>>>>> .plt_addr = 0x100000006020
>>>>>>>>> .plt.got_addr = 0x100000006240
>>>>>>>>> .fini_addr = 0x10000000737c
>>>>>>>>> .rodata_addr = 0x100000008000
>>>>>>>>> .eh_frame_hdr_addr = 0x10000000817c
>>>>>>>>> .eh_frame_addr = 0x100000008210
>>>>>>>>> .gcc_except_table_addr = 0x100000008530
>>>>>>>>> .init_array_addr = 0x100000009c60
>>>>>>>>> .fini_array_addr = 0x100000009c70
>>>>>>>>> .dynamic_addr = 0x100000009c78
>>>>>>>>> .got_addr = 0x100000009e98
>>>>>>>>> .data_addr = 0x10000000a000
>>>>>>>>> .bss_addr = 0x10000000a010
>>>>>>>>> .comment_addr = 0x100000004000
>>>>>>>>> .debug_aranges_addr = 0x100000004000
>>>>>>>>> .debug_info_addr = 0x100000004000
>>>>>>>>> .debug_abbrev_addr = 0x100000004000
>>>>>>>>> --Type <RET> for more, q to quit, c to continue without paging--c
>>>>>>>>> .debug_line_addr = 0x100000004000
>>>>>>>>> .debug_str_addr = 0x100000004000
>>>>>>>>> .debug_loc_addr = 0x100000004000
>>>>>>>>> .debug_ranges_addr = 0x100000004000
>>>>>>>>> .symtab_addr = 0x100000004000
>>>>>>>>> .strtab_addr = 0x100000004000
>>>>>>>>> warning: section .comment not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_aranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_info not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_abbrev not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_line not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_str not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_loc not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .debug_ranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .symtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> warning: section .strtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>>>>>>>>> manifest.find_file: path=/libzfs.so, found file=libzfs.so
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so 0x10000000b000
>>>>>>>>> add symbol table from file 
>>>>>>>>> "/home/matthew/osv/build/release.x64/libzfs.so" at
>>>>>>>>> .text_addr = 0x1000000178f0
>>>>>>>>> .hash_addr = 0x10000000b200
>>>>>>>>> .gnu.hash_addr = 0x10000000c318
>>>>>>>>> .dynsym_addr = 0x10000000cd78
>>>>>>>>> .dynstr_addr = 0x100000010300
>>>>>>>>> .gnu.version_addr = 0x1000000124e2
>>>>>>>>> .gnu.version_r_addr = 0x100000012958
>>>>>>>>> .rela.dyn_addr = 0x1000000129b8
>>>>>>>>> .rela.plt_addr = 0x1000000134b0
>>>>>>>>> .init_addr = 0x100000016000
>>>>>>>>> .plt_addr = 0x100000016020
>>>>>>>>> .plt.got_addr = 0x1000000178e0
>>>>>>>>> .fini_addr = 0x1000000340a0
>>>>>>>>> .rodata_addr = 0x100000035000
>>>>>>>>> .eh_frame_hdr_addr = 0x100000039768
>>>>>>>>> .eh_frame_addr = 0x10000003a470
>>>>>>>>> .init_array_addr = 0x100000040010
>>>>>>>>> .fini_array_addr = 0x100000040018
>>>>>>>>> .data.rel.ro_addr = 0x100000040020
>>>>>>>>> .dynamic_addr = 0x1000000400e0
>>>>>>>>> .got_addr = 0x1000000402f0
>>>>>>>>> .data_addr = 0x100000041000
>>>>>>>>> .bss_addr = 0x100000041820
>>>>>>>>> .comment_addr = 0x10000000b000
>>>>>>>>> .debug_aranges_addr = 0x10000000b000
>>>>>>>>> .debug_info_addr = 0x10000000b000
>>>>>>>>> .debug_abbrev_addr = 0x10000000b000
>>>>>>>>> .debug_line_addr = 0x10000000b000
>>>>>>>>> .debug_str_addr = 0x10000000b000
>>>>>>>>> .debug_loc_addr = 0x10000000b000
>>>>>>>>> .debug_ranges_addr = 0x10000000b000
>>>>>>>>> .symtab_addr = 0x10000000b000
>>>>>>>>> .strtab_addr = 0x10000000b000
>>>>>>>>> warning: section .comment not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_aranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_info not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_abbrev not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_line not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_str not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_loc not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .debug_ranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .symtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> warning: section .strtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libzfs.so
>>>>>>>>> manifest.find_file: path=/libuutil.so, found file=libuutil.so
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so 0x100000043000
>>>>>>>>> add symbol table from file 
>>>>>>>>> "/home/matthew/osv/build/release.x64/libuutil.so" at
>>>>>>>>> .text_addr = 0x1000000463c0
>>>>>>>>> .hash_addr = 0x100000043200
>>>>>>>>> .gnu.hash_addr = 0x100000043640
>>>>>>>>> .dynsym_addr = 0x1000000438f8
>>>>>>>>> .dynstr_addr = 0x100000044600
>>>>>>>>> .gnu.version_addr = 0x100000044da4
>>>>>>>>> .gnu.version_r_addr = 0x100000044ec0
>>>>>>>>> .rela.dyn_addr = 0x100000044f00
>>>>>>>>> .rela.plt_addr = 0x100000045068
>>>>>>>>> .init_addr = 0x100000046000
>>>>>>>>> .plt_addr = 0x100000046020
>>>>>>>>> .plt.got_addr = 0x1000000463b0
>>>>>>>>> .fini_addr = 0x100000049adc
>>>>>>>>> .rodata_addr = 0x10000004a000
>>>>>>>>> .eh_frame_hdr_addr = 0x10000004ac84
>>>>>>>>> .eh_frame_addr = 0x10000004af90
>>>>>>>>> .init_array_addr = 0x10000004dbd8
>>>>>>>>> .fini_array_addr = 0x10000004dbe0
>>>>>>>>> .dynamic_addr = 0x10000004dbe8
>>>>>>>>> .got_addr = 0x10000004dde8
>>>>>>>>> .data_addr = 0x10000004e000
>>>>>>>>> .bss_addr = 0x10000004e260
>>>>>>>>> .comment_addr = 0x100000043000
>>>>>>>>> .debug_aranges_addr = 0x100000043000
>>>>>>>>> .debug_info_addr = 0x100000043000
>>>>>>>>> .debug_abbrev_addr = 0x100000043000
>>>>>>>>> .debug_line_addr = 0x100000043000
>>>>>>>>> .debug_str_addr = 0x100000043000
>>>>>>>>> .debug_loc_addr = 0x100000043000
>>>>>>>>> .debug_ranges_addr = 0x100000043000
>>>>>>>>> .symtab_addr = 0x100000043000
>>>>>>>>> .strtab_addr = 0x100000043000
>>>>>>>>> warning: section .comment not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_aranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_info not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_abbrev not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_line not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_str not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_loc not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .debug_ranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .symtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> warning: section .strtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/libuutil.so
>>>>>>>>> manifest.find_file: path=/usr/lib/libgcc_s.so.1, found 
>>>>>>>>> file=%(libgcc_s_dir)s/libgcc_s.so.1
>>>>>>>>> ERROR: Unable to locate object file for: /usr/lib/libgcc_s.so.1 
>>>>>>>>> 0x10000004f000
>>>>>>>>> manifest.find_file: path=/zpool.so, found file=zpool.so
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so 0x100000069000
>>>>>>>>> add symbol table from file 
>>>>>>>>> "/home/matthew/osv/build/release.x64/zpool.so" at
>>>>>>>>> .text_addr = 0x10000006ebc0
>>>>>>>>> .hash_addr = 0x100000069200
>>>>>>>>> .gnu.hash_addr = 0x100000069868
>>>>>>>>> .dynsym_addr = 0x100000069968
>>>>>>>>> .dynstr_addr = 0x10000006ad18
>>>>>>>>> .gnu.version_addr = 0x10000006b958
>>>>>>>>> .gnu.version_r_addr = 0x10000006bb00
>>>>>>>>> .rela.dyn_addr = 0x10000006bb30
>>>>>>>>> .rela.plt_addr = 0x10000006c208
>>>>>>>>> .init_addr = 0x10000006e000
>>>>>>>>> .plt_addr = 0x10000006e020
>>>>>>>>> .plt.got_addr = 0x10000006ebb0
>>>>>>>>> .fini_addr = 0x10000007a420
>>>>>>>>> .rodata_addr = 0x10000007b000
>>>>>>>>> .eh_frame_hdr_addr = 0x10000007fd30
>>>>>>>>> .eh_frame_addr = 0x100000080028
>>>>>>>>> .init_array_addr = 0x100000082758
>>>>>>>>> .fini_array_addr = 0x100000082760
>>>>>>>>> .dynamic_addr = 0x100000082768
>>>>>>>>> .got_addr = 0x100000082978
>>>>>>>>> .data_addr = 0x100000083000
>>>>>>>>> .bss_addr = 0x100000083360
>>>>>>>>> .comment_addr = 0x100000069000
>>>>>>>>> .debug_aranges_addr = 0x100000069000
>>>>>>>>> .debug_info_addr = 0x100000069000
>>>>>>>>> .debug_abbrev_addr = 0x100000069000
>>>>>>>>> .debug_line_addr = 0x100000069000
>>>>>>>>> .debug_str_addr = 0x100000069000
>>>>>>>>> .debug_loc_addr = 0x100000069000
>>>>>>>>> .debug_ranges_addr = 0x100000069000
>>>>>>>>> .symtab_addr = 0x100000069000
>>>>>>>>> .strtab_addr = 0x100000069000
>>>>>>>>> warning: section .comment not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_aranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_info not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_abbrev not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_line not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_str not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_loc not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .debug_ranges not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .symtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> warning: section .strtab not found in 
>>>>>>>>> /home/matthew/osv/build/release.x64/zpool.so
>>>>>>>>> (gdb) osv mmap
>>>>>>>>> 0x0000000000000000 0x0000000000000000 [0.0 kB] flags=none perm=none
>>>>>>>>> 0x0000100000000000 0x0000100000001000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00000000 path=/libvdso.so
>>>>>>>>> 0x0000100000001000 0x0000100000002000 [4.0 kB] flags=fmF perm=rx 
>>>>>>>>> offset=0x00001000 path=/libvdso.so
>>>>>>>>> 0x0000100000002000 0x0000100000003000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00002000 path=/libvdso.so
>>>>>>>>> 0x0000100000003000 0x0000100000004000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00002000 path=/libvdso.so
>>>>>>>>> 0x0000100000004000 0x0000100000006000 [8.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00000000 path=/tools/mkfs.so
>>>>>>>>> 0x0000100000006000 0x0000100000008000 [8.0 kB] flags=fmF perm=rx 
>>>>>>>>> offset=0x00002000 path=/tools/mkfs.so
>>>>>>>>> 0x0000100000008000 0x0000100000009000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00004000 path=/tools/mkfs.so
>>>>>>>>> 0x0000100000009000 0x000010000000a000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00004000 path=/tools/mkfs.so
>>>>>>>>> 0x000010000000a000 0x000010000000b000 [4.0 kB] flags=fmF perm=rw 
>>>>>>>>> offset=0x00005000 path=/tools/mkfs.so
>>>>>>>>> 0x000010000000b000 0x0000100000016000 [44.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00000000 path=/libzfs.so
>>>>>>>>> 0x0000100000016000 0x0000100000035000 [124.0 kB] flags=fmF perm=rx 
>>>>>>>>> offset=0x0000b000 path=/libzfs.so
>>>>>>>>> 0x0000100000035000 0x000010000003f000 [40.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x0002a000 path=/libzfs.so
>>>>>>>>> 0x0000100000040000 0x0000100000041000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00034000 path=/libzfs.so
>>>>>>>>> 0x0000100000041000 0x0000100000042000 [4.0 kB] flags=fmF perm=rw 
>>>>>>>>> offset=0x00035000 path=/libzfs.so
>>>>>>>>> 0x0000100000042000 0x0000100000043000 [4.0 kB] flags=f perm=rw 
>>>>>>>>> 0x0000100000043000 0x0000100000046000 [12.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00000000 path=/libuutil.so
>>>>>>>>> 0x0000100000046000 0x000010000004a000 [16.0 kB] flags=fmF perm=rx 
>>>>>>>>> offset=0x00003000 path=/libuutil.so
>>>>>>>>> 0x000010000004a000 0x000010000004c000 [8.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00007000 path=/libuutil.so
>>>>>>>>> 0x000010000004d000 0x000010000004e000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00009000 path=/libuutil.so
>>>>>>>>> 0x000010000004e000 0x000010000004f000 [4.0 kB] flags=fmF perm=rw 
>>>>>>>>> offset=0x0000a000 path=/libuutil.so
>>>>>>>>> 0x000010000004f000 0x0000100000052000 [12.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00000000 path=/usr/lib/libgcc_s.so.1
>>>>>>>>> 0x0000100000052000 0x0000100000063000 [68.0 kB] flags=fmF perm=rx 
>>>>>>>>> offset=0x00003000 path=/usr/lib/libgcc_s.so.1
>>>>>>>>> 0x0000100000063000 0x0000100000067000 [16.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00014000 path=/usr/lib/libgcc_s.so.1
>>>>>>>>> 0x0000100000067000 0x0000100000068000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00017000 path=/usr/lib/libgcc_s.so.1
>>>>>>>>> 0x0000100000068000 0x0000100000069000 [4.0 kB] flags=fmF perm=rw 
>>>>>>>>> offset=0x00018000 path=/usr/lib/libgcc_s.so.1
>>>>>>>>> 0x0000100000069000 0x000010000006e000 [20.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00000000 path=/zpool.so
>>>>>>>>> 0x000010000006e000 0x000010000007b000 [52.0 kB] flags=fmF perm=rx 
>>>>>>>>> offset=0x00005000 path=/zpool.so
>>>>>>>>> 0x000010000007b000 0x0000100000082000 [28.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00012000 path=/zpool.so
>>>>>>>>> 0x0000100000082000 0x0000100000083000 [4.0 kB] flags=fmF perm=r 
>>>>>>>>> offset=0x00018000 path=/zpool.so
>>>>>>>>> 0x0000100000083000 0x0000100000084000 [4.0 kB] flags=fmF perm=rw 
>>>>>>>>> offset=0x00019000 path=/zpool.so
>>>>>>>>> 0x0000100000084000 0x0000100000086000 [8.0 kB] flags=f perm=rw 
>>>>>>>>> 0x0000200000000000 0x0000200000001000 [4.0 kB] flags=p perm=none
>>>>>>>>> 0x0000200000001000 0x0000200000002000 [4.0 kB] flags=p perm=none
>>>>>>>>> 0x0000200000002000 0x0000200000101000 [1020.0 kB] flags=p perm=rw 
>>>>>>>>> 0x0000200000101000 0x0000200000102000 [4.0 kB] flags=p perm=none
>>>>>>>>> 0x0000200000102000 0x0000200000201000 [1020.0 kB] flags=p perm=rw 
>>>>>>>>> 0x0000800000000000 0x0000800000000000 [0.0 kB] flags=none perm=none
>>>>>>>>> (gdb) bt
>>>>>>>>> #0 abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: 
>>>>>>>>> %d)\n") at runtime.cc:105
>>>>>>>>> #1 0x000000004023c75b in __assert_fail (expr=expr@entry=0x406742f8 
>>>>>>>>> "ef->rflags & processor::rflags_if", file=file@entry=0x40674325 
>>>>>>>>> "arch/x64/mmu.cc", line=line@entry=38, func=func@entry=0x4067431a 
>>>>>>>>> "page_fault") at runtime.cc:139
>>>>>>>>> #2 0x0000000040399305 in page_fault (ef=0xffff800000015048) at 
>>>>>>>>> arch/x64/arch-cpu.hh:107
>>>>>>>>> #3 <signal handler called>
>>>>>>>>> #4 0x000000004035ca19 in elf::object::symtab_len 
>>>>>>>>> (this=0xffffa00000f19a00) at core/elf.cc:983
>>>>>>>>> #5 0x000000004035cad8 in elf::object::lookup_addr 
>>>>>>>>> (this=0xffffa00000f19a00, addr=addr@entry=0x1000000254ce 
>>>>>>>>> <is_mounted+30>) 
>>>>>>>>> at core/elf.cc:1015
>>>>>>>>> #6 0x000000004035cca7 in elf::program::<lambda(const 
>>>>>>>>> elf::program::modules_list&)>::operator() (__closure=<synthetic 
>>>>>>>>> pointer>, 
>>>>>>>>> __closure=<synthetic pointer>, ml=...) at core/elf.cc:1620
>>>>>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void 
>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> > (f=..., 
>>>>>>>>> this=0xffffa00000097e70) at include/osv/elf.hh:702
>>>>>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70, 
>>>>>>>>> addr=addr@entry=0x1000000254ce <is_mounted+30>) at core/elf.cc:1617
>>>>>>>>> #9 0x00000000404367cc in osv::lookup_name_demangled 
>>>>>>>>> (addr=addr@entry=0x1000000254ce <is_mounted+30>, 
>>>>>>>>> buf=buf@entry=0xffff8000012156d0 "???+19929103", len=len@entry=1024) 
>>>>>>>>> at 
>>>>>>>>> core/demangle.cc:47
>>>>>>>>> #10 0x000000004023c540 in print_backtrace () at runtime.cc:85
>>>>>>>>> #11 0x000000004023c714 in abort (fmt=fmt@entry=0x40645aff 
>>>>>>>>> "Aborted\n") at runtime.cc:121
>>>>>>>>> #12 0x0000000040202989 in abort () at runtime.cc:98
>>>>>>>>> #13 0x0000000040345934 in mmu::vm_sigsegv (ef=0xffff800001216068, 
>>>>>>>>> addr=<optimized out>) at core/mmu.cc:1314
>>>>>>>>> #14 mmu::vm_sigsegv (addr=<optimized out>, ef=0xffff800001216068) 
>>>>>>>>> at core/mmu.cc:1308
>>>>>>>>> #15 0x000000004034782f in mmu::vm_fault 
>>>>>>>>> (addr=addr@entry=17592186309800, ef=ef@entry=0xffff800001216068) at 
>>>>>>>>> core/mmu.cc:1328
>>>>>>>>> #16 0x00000000403992a3 in page_fault (ef=0xffff800001216068) at 
>>>>>>>>> arch/x64/mmu.cc:42
>>>>>>>>> #17 <signal handler called>
>>>>>>>>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot 
>>>>>>>>> (this=this@entry=0xffffa00000f19a00, sym=..., 
>>>>>>>>> addr=addr@entry=0x100000040ca8 <[email protected]>, 
>>>>>>>>> addend=addend@entry=0) 
>>>>>>>>> at arch/x64/arch-elf.cc:172
>>>>>>>>> #19 0x0000000040361004 in elf::object::resolve_pltgot 
>>>>>>>>> (this=0xffffa00000f19a00, index=<optimized out>) at core/elf.cc:843
>>>>>>>>> #20 0x0000000040361229 in elf_resolve_pltgot (index=308, 
>>>>>>>>> obj=0xffffa00000f19a00) at core/elf.cc:1860
>>>>>>>>> #21 0x0000000040397d50 in __elf_resolve_pltgot () at 
>>>>>>>>> arch/x64/elf-dl.S:47
>>>>>>>>> #22 0x00001000000254cf in is_mounted (zfs_hdl=0x134, 
>>>>>>>>> special=<optimized out>, where=0x403f3377 <malloc(size_t)+71>) at 
>>>>>>>>> bsd/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_mount.c:214
>>>>>>>>> #23 0xffff900000a99000 in ?? ()
>>>>>>>>> #24 0x0000000000000000 in ?? ()
>>>>>>>>>
>>>>>>>>> On Tuesday, December 8, 2020 at 8:53:52 AM UTC-7 
>>>>>>>>> [email protected] wrote:
>>>>>>>>>
>>>>>>>>>> It would be also nice to understand if we are crashing on the 1st 
>>>>>>>>>> arch_relocate_jump_slot() for libfzs.so or is it a specific 
>>>>>>>>>> JUMP_SLOT that 
>>>>>>>>>> causes this crash? 
>>>>>>>>>>
>>>>>>>>>> On Tuesday, December 8, 2020 at 10:39:06 AM UTC-5 Waldek Kozaczuk 
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> After you connect with gdb can you run 'osv mmap' and send us 
>>>>>>>>>>> the output. Make sure you run 'osv syms' before it and dump 
>>>>>>>>>>> backtrace 
>>>>>>>>>>> after. Please see 
>>>>>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv for 
>>>>>>>>>>> any details.
>>>>>>>>>>>
>>>>>>>>>>> BTW can you build and run OSv ZFS image on the host without NIX? 
>>>>>>>>>>> As I understand NIX is really just a layer on top of any Linux 
>>>>>>>>>>> distribution, no? I am afraid I do not still understand what 
>>>>>>>>>>> exactly NiX is 
>>>>>>>>>>> I guess.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Monday, December 7, 2020 at 2:58:40 PM UTC-5 Matthew 
>>>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>>>
>>>>>>>>>>>> (gdb) frame 18
>>>>>>>>>>>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot 
>>>>>>>>>>>> (this=this@entry=0xffffa0000110fa00, sym=..., 
>>>>>>>>>>>>     addr=addr@entry=0x100000040ca8, addend=addend@entry=0) at 
>>>>>>>>>>>> arch/x64/arch-elf.cc:172
>>>>>>>>>>>> 172            *static_cast<void**>(addr) = 
>>>>>>>>>>>> sym.relocated_addr();
>>>>>>>>>>>> (gdb) print _pathname
>>>>>>>>>>>> $14 = {static npos = 18446744073709551615, 
>>>>>>>>>>>>   _M_dataplus = {<std::allocator<char>> = 
>>>>>>>>>>>> {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data 
>>>>>>>>>>>> fields>}, 
>>>>>>>>>>>>     _M_p = 0xffffa0000110fa30 "/libzfs.so"}, _M_string_length = 
>>>>>>>>>>>> 10, {
>>>>>>>>>>>>     _M_local_buf = "/libzfs.so\000\000\000\000\000", 
>>>>>>>>>>>> _M_allocated_capacity = 3347131623889529903}}
>>>>>>>>>>>>
>>>>>>>>>>>> Also been wondering if nix using nonstandard paths is causing 
>>>>>>>>>>>> problems, like for libc:
>>>>>>>>>>>> [nix-shell:~/osv/build/release]$ ldd libzfs.so 
>>>>>>>>>>>>     linux-vdso.so.1 (0x00007ffcedbb9000)
>>>>>>>>>>>>     libuutil.so => not found
>>>>>>>>>>>>     libc.so.6 => 
>>>>>>>>>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libc.so.6
>>>>>>>>>>>>  
>>>>>>>>>>>> (0x00007f7594f38000)
>>>>>>>>>>>>    
>>>>>>>>>>>>  
>>>>>>>>>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib64/ld-linux-x86-64.so.2
>>>>>>>>>>>>  
>>>>>>>>>>>> (0x00007f7595131000)
>>>>>>>>>>>> On Sunday, December 6, 2020 at 8:43:10 AM UTC-7 
>>>>>>>>>>>> [email protected] wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> It might be easier to simply print '_pathname' value if you 
>>>>>>>>>>>>> switch to the right frame in gdb. It would be nice to confirm 
>>>>>>>>>>>>> that the 
>>>>>>>>>>>>> problem we have is with zpool.so and that might lead to 
>>>>>>>>>>>>> understanding why 
>>>>>>>>>>>>> this crash happens. Maybe the is something wrong with building 
>>>>>>>>>>>>> zpool.so.
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW based on this fragment of the stacktrace:
>>>>>>>>>>>>>
>>>>>>>>>>>>> #6  0x000000004035cb07 in elf::program::<lambda(const 
>>>>>>>>>>>>> elf::program::modules_list&)>::operator() (
>>>>>>>>>>>>>     __closure=<synthetic pointer>, __closure=<synthetic 
>>>>>>>>>>>>> pointer>, ml=...) at core/elf.cc:1620
>>>>>>>>>>>>> #7  elf::program::with_modules<elf::program::lookup_addr(void 
>>>>>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> >
>>>>>>>>>>>>>     (f=..., this=0xffffa00000097e70) at include/osv/elf.hh:702
>>>>>>>>>>>>> #8  elf::program::lookup_addr (this=0xffffa00000097e70, 
>>>>>>>>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617
>>>>>>>>>>>>> #9  0x00000000404357cc in osv::lookup_name_demangled 
>>>>>>>>>>>>> (addr=addr@entry=0x1000000254ce,
>>>>>>>>>>>>>     buf=buf@entry=0xffff8000012146d0 "???+19630095", 
>>>>>>>>>>>>> len=len@entry=1024) at core/demangle.cc:47
>>>>>>>>>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems we have a bug (or need of improvement) in 
>>>>>>>>>>>>> print_backtrace() to make it NOT try to demangle names like 
>>>>>>>>>>>>> "???+19630095" 
>>>>>>>>>>>>> which causes follow-up fault.
>>>>>>>>>>>>>
>>>>>>>>>>>>> At the same time, it is strange that we crash at line 983 
>>>>>>>>>>>>> which seems to indicate something goes wrong when processing 
>>>>>>>>>>>>> zpool.so.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  981     if (dynamic_exists(DT_HASH)) {
>>>>>>>>>>>>>
>>>>>>>>>>>>>  982         auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH);
>>>>>>>>>>>>>
>>>>>>>>>>>>>  *983         return hashtab[1];*
>>>>>>>>>>>>>
>>>>>>>>>>>>>  984     }
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sunday, December 6, 2020 at 10:06:21 AM UTC-5 Waldek 
>>>>>>>>>>>>> Kozaczuk wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you run the ROFS image you built? Also as I understand it 
>>>>>>>>>>>>>> NIX is a package manager but what Linux distribution are you 
>>>>>>>>>>>>>> using?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As far as ZFS goes could you enable ELF debugging - change 
>>>>>>>>>>>>>> this line:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> conf-debug_elf=0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> conf-debug_elf=1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In conf/base.mk, delete core/elf.o and force rebuild the 
>>>>>>>>>>>>>> kernel. I think you may also need to change the script 
>>>>>>>>>>>>>> upload_manifest.py 
>>>>>>>>>>>>>> to peeped ‘—verbose’ to the command line with cpiod.so
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It should show more info about elf loading. It may still be 
>>>>>>>>>>>>>> necessary to add extra printouts to capture which exact elf it 
>>>>>>>>>>>>>> is crashing 
>>>>>>>>>>>>>> on in arch_relocate_jump(). 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In worst case I would need a copy of your loader-stripped.elf 
>>>>>>>>>>>>>> and possibly all the other files like cpiod.so, zfs.so that go 
>>>>>>>>>>>>>> into the 
>>>>>>>>>>>>>> bootfs part of the image. 
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Waldek
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Dec 5, 2020 at 19:31 Matthew Kenigsberg <
>>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> After forcing it to use the right path for libz.so.1, it's 
>>>>>>>>>>>>>>> working with rofs, but still having the same issue when using 
>>>>>>>>>>>>>>> zfs, even 
>>>>>>>>>>>>>>> after I correct the path for libz.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Saturday, December 5, 2020 at 5:18:37 PM UTC-7 Matthew 
>>>>>>>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> gcc version 9.3.0 (GCC)
>>>>>>>>>>>>>>>> QEMU emulator version 5.1.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Running with fs=rofs I get the error:
>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 369, in <module>
>>>>>>>>>>>>>>>>     main()
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 366, in main
>>>>>>>>>>>>>>>>     gen_image(outfile, manifest)
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 269, in gen_image
>>>>>>>>>>>>>>>>     system_structure_block, bytes_written = write_fs(fp, 
>>>>>>>>>>>>>>>> manifest)
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 246, in write_fs
>>>>>>>>>>>>>>>>     count, directory_entries_index = write_dir(fp, 
>>>>>>>>>>>>>>>> manifest.get(''), '', manifest)
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 207, in write_dir
>>>>>>>>>>>>>>>>     count, directory_entries_index = write_dir(fp, val, 
>>>>>>>>>>>>>>>> dirpath + '/' + entry, manifest)
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 207, in write_dir
>>>>>>>>>>>>>>>>     count, directory_entries_index = write_dir(fp, val, 
>>>>>>>>>>>>>>>> dirpath + '/' + entry, manifest)
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 222, in write_dir
>>>>>>>>>>>>>>>>     inode.count = write_file(fp, val)
>>>>>>>>>>>>>>>>   File "/home/matthew/osv/scripts/gen-rofs-img.py", line 
>>>>>>>>>>>>>>>> 164, in write_file
>>>>>>>>>>>>>>>>     with open(path, 'rb') as f:
>>>>>>>>>>>>>>>> FileNotFoundError: [Errno 2] No such file or directory: 
>>>>>>>>>>>>>>>> 'libz.so.1'
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think that's from this line in usr.manifest?
>>>>>>>>>>>>>>>> /usr/lib/libz.so.1: libz.so.1
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Don't have zlib in the manifest without fs=rofs, and I 
>>>>>>>>>>>>>>>> think zpool uses it?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Looking into it...
>>>>>>>>>>>>>>>> On Saturday, December 5, 2020 at 4:36:20 PM UTC-7 
>>>>>>>>>>>>>>>> [email protected] wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I can not reproduce it on Ubuntu 20.20 neither Fedora 33. 
>>>>>>>>>>>>>>>>> Here is the code fragment where it happens:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 169 bool object::arch_relocate_jump_slot(symbol_module& 
>>>>>>>>>>>>>>>>> sym, void *addr, Elf64_Sxword addend)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 170 {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 171     if (sym.symbol) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 172         *static_cast<void**>(addr) = 
>>>>>>>>>>>>>>>>> sym.relocated_addr();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 173         return true;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 174     } else {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 175         return false;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 176     }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 177 }
>>>>>>>>>>>>>>>>> It looks like writing at the addr 0x100000040ca8 in line 
>>>>>>>>>>>>>>>>> 172 caused the fault. Why?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And then the 2nd page fault in the gdb backtrace as the 
>>>>>>>>>>>>>>>>> 1st one was being handled (not sure if that is a bug or just 
>>>>>>>>>>>>>>>>> a state of 
>>>>>>>>>>>>>>>>> loading of a program).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 981     if (dynamic_exists(DT_HASH)) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>  982         auto hashtab = 
>>>>>>>>>>>>>>>>> dynamic_ptr<Elf64_Word>(DT_HASH);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>  983         return hashtab[1];
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>  984     }
>>>>>>>>>>>>>>>>> Is something wrong with the elf files cpiod.so, mkfs.so or 
>>>>>>>>>>>>>>>>> zfs.so or something?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you try to do the same with ROFS?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> fs=rofs
>>>>>>>>>>>>>>>>> On Saturday, December 5, 2020 at 5:44:12 PM UTC-5 Matthew 
>>>>>>>>>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Struggling to get scripts/build to run on NixOS because 
>>>>>>>>>>>>>>>>>> I'm getting a page fault. NixOS does keep shared libraries 
>>>>>>>>>>>>>>>>>> in nonstandard 
>>>>>>>>>>>>>>>>>> locations, not sure if that's breaking something. More 
>>>>>>>>>>>>>>>>>> details below, but 
>>>>>>>>>>>>>>>>>> any ideas?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As far as I can tell, the error is caused by 
>>>>>>>>>>>>>>>>>> tools/mkfs/mkfs.cc:71:
>>>>>>>>>>>>>>>>>>     run_cmd("/zpool.so", zpool_args);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The error from scripts/build:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> OSv v0.55.0-145-g97f17a7a
>>>>>>>>>>>>>>>>>> eth0: 192.168.122.15
>>>>>>>>>>>>>>>>>> Booted up in 154.38 ms
>>>>>>>>>>>>>>>>>> Cmdline: /tools/mkfs.so; /tools/cpiod.so --prefix 
>>>>>>>>>>>>>>>>>> /zfs/zfs/; /zfs.so set compression=off osv
>>>>>>>>>>>>>>>>>> Running mkfs...
>>>>>>>>>>>>>>>>>> page fault outside application, addr: 0x0000100000040ca8
>>>>>>>>>>>>>>>>>> [registers]
>>>>>>>>>>>>>>>>>> RIP: 0x000000004039c25a 
>>>>>>>>>>>>>>>>>> <elf::object::arch_relocate_jump_slot(elf::symbol_module&, 
>>>>>>>>>>>>>>>>>> void*, long)+26>
>>>>>>>>>>>>>>>>>> RFL: 0x0000000000010202  CS:  0x0000000000000008  SS:  
>>>>>>>>>>>>>>>>>> 0x0000000000000010
>>>>>>>>>>>>>>>>>> RAX: 0x000010000007a340  RBX: 0x0000100000040ca8  RCX: 
>>>>>>>>>>>>>>>>>> 0x000010000006abb0  RDX: 0x0000000000000002
>>>>>>>>>>>>>>>>>> RSI: 0x00002000001f6f70  RDI: 0xffffa00001058c00  RBP: 
>>>>>>>>>>>>>>>>>> 0x00002000001f6f30  R8:  0xffffa00000a68460
>>>>>>>>>>>>>>>>>> R9:  0xffffa00000f18da0  R10: 0x0000000000000000  R11: 
>>>>>>>>>>>>>>>>>> 0x00000000409dd380  R12: 0xffffa00000f18c00
>>>>>>>>>>>>>>>>>> R13: 0xffffa00000f18da0  R14: 0x0000000000000000  R15: 
>>>>>>>>>>>>>>>>>> 0x00000000409dd380  RSP: 0x00002000001f6f20
>>>>>>>>>>>>>>>>>> Aborted
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [backtrace]
>>>>>>>>>>>>>>>>>> 0x00000000403458d3 <???+1077172435>
>>>>>>>>>>>>>>>>>> 0x00000000403477ce <mmu::vm_fault(unsigned long, 
>>>>>>>>>>>>>>>>>> exception_frame*)+350>
>>>>>>>>>>>>>>>>>> 0x0000000040398ba2 <page_fault+162>
>>>>>>>>>>>>>>>>>> 0x0000000040397a16 <???+1077508630>
>>>>>>>>>>>>>>>>>> 0x0000000040360a13 <elf::object::resolve_pltgot(unsigned 
>>>>>>>>>>>>>>>>>> int)+387>
>>>>>>>>>>>>>>>>>> 0x0000000040360c38 <elf_resolve_pltgot+56>
>>>>>>>>>>>>>>>>>> 0x000000004039764f <???+1077507663>
>>>>>>>>>>>>>>>>>> 0xffffa000012b880f <???+19630095>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Trying to get a backtrace after connecting with gdb:
>>>>>>>>>>>>>>>>>> (gdb) bt
>>>>>>>>>>>>>>>>>> #0  abort (fmt=fmt@entry=0x40644b90 "Assertion failed: %s 
>>>>>>>>>>>>>>>>>> (%s: %s: %d)\n") at runtime.cc:105
>>>>>>>>>>>>>>>>>> #1  0x000000004023c6fb in __assert_fail 
>>>>>>>>>>>>>>>>>> (expr=expr@entry=0x40672cf8 "ef->rflags & 
>>>>>>>>>>>>>>>>>> processor::rflags_if", 
>>>>>>>>>>>>>>>>>>     file=file@entry=0x40672d25 "arch/x64/mmu.cc", 
>>>>>>>>>>>>>>>>>> line=line@entry=38, func=func@entry=0x40672d1a "page_fault")
>>>>>>>>>>>>>>>>>>     at runtime.cc:139
>>>>>>>>>>>>>>>>>> #2  0x0000000040398c05 in page_fault 
>>>>>>>>>>>>>>>>>> (ef=0xffff800000015048) at arch/x64/arch-cpu.hh:107
>>>>>>>>>>>>>>>>>> #3  <signal handler called>
>>>>>>>>>>>>>>>>>> #4  0x000000004035c879 in elf::object::symtab_len 
>>>>>>>>>>>>>>>>>> (this=0xffffa00000f18c00) at core/elf.cc:983
>>>>>>>>>>>>>>>>>> #5  0x000000004035c938 in elf::object::lookup_addr 
>>>>>>>>>>>>>>>>>> (this=0xffffa00000f18c00, addr=addr@entry=0x1000000254ce)
>>>>>>>>>>>>>>>>>>     at core/elf.cc:1015
>>>>>>>>>>>>>>>>>> #6  0x000000004035cb07 in elf::program::<lambda(const 
>>>>>>>>>>>>>>>>>> elf::program::modules_list&)>::operator() (
>>>>>>>>>>>>>>>>>>     __closure=<synthetic pointer>, __closure=<synthetic 
>>>>>>>>>>>>>>>>>> pointer>, ml=...) at core/elf.cc:1620
>>>>>>>>>>>>>>>>>> #7  
>>>>>>>>>>>>>>>>>> elf::program::with_modules<elf::program::lookup_addr(void 
>>>>>>>>>>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> >
>>>>>>>>>>>>>>>>>>     (f=..., this=0xffffa00000097e70) at 
>>>>>>>>>>>>>>>>>> include/osv/elf.hh:702
>>>>>>>>>>>>>>>>>> #8  elf::program::lookup_addr (this=0xffffa00000097e70, 
>>>>>>>>>>>>>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617
>>>>>>>>>>>>>>>>>> #9  0x00000000404357cc in osv::lookup_name_demangled 
>>>>>>>>>>>>>>>>>> (addr=addr@entry=0x1000000254ce, 
>>>>>>>>>>>>>>>>>>     buf=buf@entry=0xffff8000012146d0 "???+19630095", 
>>>>>>>>>>>>>>>>>> len=len@entry=1024) at core/demangle.cc:47
>>>>>>>>>>>>>>>>>> #10 0x000000004023c4e0 in print_backtrace () at 
>>>>>>>>>>>>>>>>>> runtime.cc:85
>>>>>>>>>>>>>>>>>> #11 0x000000004023c6b4 in abort (fmt=fmt@entry=0x40644a9f 
>>>>>>>>>>>>>>>>>> "Aborted\n") at runtime.cc:121
>>>>>>>>>>>>>>>>>> #12 0x0000000040202989 in abort () at runtime.cc:98
>>>>>>>>>>>>>>>>>> #13 0x00000000403458d4 in mmu::vm_sigsegv 
>>>>>>>>>>>>>>>>>> (ef=0xffff800001215068, addr=<optimized out>) at 
>>>>>>>>>>>>>>>>>> core/mmu.cc:1314
>>>>>>>>>>>>>>>>>> #14 mmu::vm_sigsegv (addr=<optimized out>, 
>>>>>>>>>>>>>>>>>> ef=0xffff800001215068) at core/mmu.cc:1308
>>>>>>>>>>>>>>>>>> #15 0x00000000403477cf in mmu::vm_fault 
>>>>>>>>>>>>>>>>>> (addr=addr@entry=17592186309800, 
>>>>>>>>>>>>>>>>>> ef=ef@entry=0xffff800001215068)
>>>>>>>>>>>>>>>>>>     at core/mmu.cc:1328
>>>>>>>>>>>>>>>>>> #16 0x0000000040398ba3 in page_fault 
>>>>>>>>>>>>>>>>>> (ef=0xffff800001215068) at arch/x64/mmu.cc:42
>>>>>>>>>>>>>>>>>> #17 <signal handler called>
>>>>>>>>>>>>>>>>>> #18 0x000000004039c25a in 
>>>>>>>>>>>>>>>>>> elf::object::arch_relocate_jump_slot 
>>>>>>>>>>>>>>>>>> (this=this@entry=0xffffa00000f18c00, 
>>>>>>>>>>>>>>>>>> sym=..., 
>>>>>>>>>>>>>>>>>>     addr=addr@entry=0x100000040ca8, 
>>>>>>>>>>>>>>>>>> addend=addend@entry=0) at arch/x64/arch-elf.cc:172
>>>>>>>>>>>>>>>>>> #19 0x0000000040360a14 in elf::object::resolve_pltgot 
>>>>>>>>>>>>>>>>>> (this=0xffffa00000f18c00, index=<optimized out>)
>>>>>>>>>>>>>>>>>>     at core/elf.cc:843
>>>>>>>>>>>>>>>>>> #20 0x0000000040360c39 in elf_resolve_pltgot (index=308, 
>>>>>>>>>>>>>>>>>> obj=0xffffa00000f18c00) at core/elf.cc:1860
>>>>>>>>>>>>>>>>>> #21 0x0000000040397650 in __elf_resolve_pltgot () at 
>>>>>>>>>>>>>>>>>> arch/x64/elf-dl.S:47
>>>>>>>>>>>>>>>>>> #22 0x00001000000254cf in ?? ()
>>>>>>>>>>>>>>>>>> #23 0xffffa000012b8800 in ?? ()
>>>>>>>>>>>>>>>>>> #24 0x00002000001f74a0 in ?? ()
>>>>>>>>>>>>>>>>>> #25 0x00001000000254cf in ?? ()
>>>>>>>>>>>>>>>>>> #26 0x00002000001f7480 in ?? ()
>>>>>>>>>>>>>>>>>> #27 0x00000000403f241c in calloc (nmemb=<optimized out>, 
>>>>>>>>>>>>>>>>>> size=<optimized out>) at core/mempool.cc:1811
>>>>>>>>>>>>>>>>>> #28 0xffff900000a98000 in ?? ()
>>>>>>>>>>>>>>>>>> #29 0x0000000000000000 in ?? ()
>>>>>>>>>>>>>>>>>> On Saturday, November 28, 2020 at 1:39:46 PM UTC-7 
>>>>>>>>>>>>>>>>>> Matthew Kenigsberg wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'll send something, might take a bit before I find time 
>>>>>>>>>>>>>>>>>>> to work on it though.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Matthew
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Saturday, November 28, 2020 at 1:11:11 PM UTC-7 Roman 
>>>>>>>>>>>>>>>>>>> Shaposhnik wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Nov 24, 2020 at 8:03 AM Waldek Kozaczuk <
>>>>>>>>>>>>>>>>>>>> [email protected]> wrote: 
>>>>>>>>>>>>>>>>>>>> > 
>>>>>>>>>>>>>>>>>>>> > Hey, 
>>>>>>>>>>>>>>>>>>>> > 
>>>>>>>>>>>>>>>>>>>> > Send a patch with a new app that could demonstrate 
>>>>>>>>>>>>>>>>>>>> it, please, if you can. I would like to see it. Sounds 
>>>>>>>>>>>>>>>>>>>> like a nice 
>>>>>>>>>>>>>>>>>>>> improvement. 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> FWIW: I'd love to see it too -- been meaning to play 
>>>>>>>>>>>>>>>>>>>> with Nix and this 
>>>>>>>>>>>>>>>>>>>> gives me a perfect excuse ;-) 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks, 
>>>>>>>>>>>>>>>>>>>> Roman. 
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>> You received this message because you are subscribed to a 
>>>>>>>>>>>>>>> topic in the Google Groups "OSv Development" group.
>>>>>>>>>>>>>>> To unsubscribe from this topic, visit 
>>>>>>>>>>>>>>> https://groups.google.com/d/topic/osv-dev/rhjHPr7OBEw/unsubscribe
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>> To unsubscribe from this group and all its topics, send an 
>>>>>>>>>>>>>>> email to [email protected].
>>>>>>>>>>>>>>> To view this discussion on the web visit 
>>>>>>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com
>>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/34b636e2-445e-49cb-99d5-446c55f69704n%40googlegroups.com.

Reply via email to