Not completely sure where libgcc_s.so.1 is coming from, but I uploaded what I have in /nix/store/vran8acwir59772hj4vscr7zribvp7l5-gcc-9.3.0-lib/lib/libgcc_s.so.1: https://drive.google.com/drive/folders/1rM6g-FrzwFpuHr2wX9-J21DzSjyQXGg2
Get a different error if I comment out core/elf.cc:1429: (gdb) bt #0 0x000000004039eef2 in processor::cli_hlt () at arch/x64/processor.hh:247 #1 arch::halt_no_interrupts () at arch/x64/arch.hh:48 #2 osv::halt () at arch/x64/power.cc:26 #3 0x000000004023c73f in abort (fmt=fmt@entry=0x40645aff "Aborted\n") at runtime.cc:132 #4 0x0000000040202989 in abort () at runtime.cc:98 #5 0x0000000040218943 in osv::generate_signal (siginfo=..., ef=0xffff80000191c068) at libc/signal.cc:124 #6 0x00000000404745ff in osv::handle_mmap_fault (addr=<optimized out>, sig=<optimized out>, ef=<optimized out>) at libc/signal.cc:139 #7 0x0000000040347872 in mmu::vm_fault (addr=17592187039744, addr@entry=17592187040376, ef=ef@entry=0xffff80000191c068) at core/mmu.cc:1336 #8 0x00000000403992e3 in page_fault (ef=0xffff80000191c068) at arch/x64/mmu.cc:42 #9 <signal handler called> #10 0x00001000001554c4 in usage (requested=requested@entry=false) at bsd/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:424 #11 0x0000100000152025 in main (argc=5, argv=0xffffa00000f19400) at bsd/cddl/contrib/opensolaris/cmd/zfs/zfs_main.c:6676 #12 0x000000004043c311 in osv::application::run_main (this=0xffffa00000ee0c10) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/stl_vector.h:915 #13 0x000000004022452f in osv::application::main (this=0xffffa00000ee0c10) at core/app.cc:320 #14 0x000000004043c539 in osv::application::<lambda(void*)>::operator() (__closure=0x0, app=<optimized out>) at core/app.cc:233 #15 osv::application::<lambda(void*)>::_FUN(void *) () at core/app.cc:235 #16 0x0000000040470d58 in pthread_private::pthread::<lambda()>::operator() (__closure=0xffffa00001067f00) at libc/pthread.cc:115 #17 std::_Function_handler<void(), pthread_private::pthread::pthread(void* (*)(void*), void*, sigset_t, const pthread_private::thread_attr*)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_function.h:300 #18 0x00000000404074fd in sched::thread_main_c (t=0xffff800001917040) at arch/x64/arch-switch.hh:325 #19 0x00000000403990d3 in thread_main () at arch/x64/entry.S:113 On Tuesday, December 8, 2020 at 2:34:19 PM UTC-7 [email protected] wrote: > I wonder if we have a bug in core/elf.cc::fix_permissions() or logic > around. And we might be making the wrong part of the mapping readable based > on GNU_RELRO header. I wonder if you are able to create ZFS image by > temporarily commenting out the line 1429 of core/elf.cc: > > ef->fix_permissions(); > Also, would it possible to get copies of those binaries: > /libenviron.so: libenviron.so > /libvdso.so: libvdso.so > /zpool.so: zpool.so > /libzfs.so: libzfs.so > /libuutil.so: libuutil.so > /zfs.so: zfs.so > /tools/mkfs.so: tools/mkfs/mkfs.so > /tools/cpiod.so: tools/cpiod/cpiod.so > /tools/mount-fs.so: tools/mount/mount-fs.so > /tools/umount.so: tools/mount/umount.so > /usr/lib/libgcc_s.so.1: %(libgcc_s_dir)s/libgcc_s.so.1 > > Ideally, the stripped versions. That would help me to re-create the > problem and investigate further. > > On Tuesday, December 8, 2020 at 1:25:10 PM UTC-5 Matthew Kenigsberg wrote: > >> [nix-shell:~/osv]$ readelf -l build/release/libzfs-stripped.so >> >> Elf file type is DYN (Shared object file) >> Entry point 0xc8f0 >> There are 8 program headers, starting at offset 64 >> >> Program Headers: >> Type Offset VirtAddr PhysAddr >> FileSiz MemSiz Flags Align >> LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000 >> 0x000000000000a9b8 0x000000000000a9b8 R 0x1000 >> LOAD 0x000000000000b000 0x000000000000b000 0x000000000000b000 >> 0x000000000001e0a9 0x000000000001e0a9 R E 0x1000 >> LOAD 0x000000000002a000 0x000000000002a000 0x000000000002a000 >> 0x00000000000093a0 0x00000000000093a0 R 0x1000 >> LOAD 0x0000000000034010 0x0000000000035010 0x0000000000035010 >> 0x0000000000001810 0x0000000000002c20 RW 0x1000 >> DYNAMIC 0x00000000000340e0 0x00000000000350e0 0x00000000000350e0 >> 0x0000000000000210 0x0000000000000210 RW 0x8 >> GNU_EH_FRAME 0x000000000002e768 0x000000000002e768 0x000000000002e768 >> 0x0000000000000d04 0x0000000000000d04 R 0x4 >> GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 >> 0x0000000000000000 0x0000000000000000 RW 0x10 >> GNU_RELRO 0x0000000000034010 0x0000000000035010 0x0000000000035010 >> 0x0000000000000ff0 0x0000000000000ff0 R 0x1 >> >> Section to Segment mapping: >> Segment Sections... >> 00 .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn >> .rela.plt >> 01 .init .plt .plt.got .text .fini >> 02 .rodata .eh_frame_hdr .eh_frame >> 03 .init_array .fini_array .data.rel.ro .dynamic .got .data .bss >> 04 .dynamic >> 05 .eh_frame_hdr >> 06 >> 07 .init_array .fini_array .data.rel.ro .dynamic .got >> >> On Tuesday, December 8, 2020 at 11:17:46 AM UTC-7 [email protected] >> wrote: >> >>> Back to why it is failing. >>> >>> Based on what you sent us: >>> .. >>> 0x000010000000b000 0x0000100000016000 [44.0 kB] flags=fmF perm=r >>> offset=0x00000000 path=/libzfs.so >>> 0x0000100000016000 0x0000100000035000 [124.0 kB] flags=fmF perm=rx >>> offset=0x0000b000 path=/libzfs.so >>> 0x0000100000035000 0x000010000003f000 [40.0 kB] flags=fmF perm=r >>> offset=0x0002a000 path=/libzfs.so >>> >>> *0x0000100000040000 0x0000100000041000 [4.0 kB] flags=fmF perm=r >>> offset=0x00034000 path=/libzfs.so*0x0000100000041000 0x0000100000042000 >>> [4.0 kB] flags=fmF perm=rw offset=0x00035000 path=/libzfs.so >>> .. >>> >>> The page fault in arch_relocate_jump_slot() is caused by an attempt to >>> write at the address 0x100000040ca8 which falls into a 4th mapping range >>> from the above which can only be read from. So that is the permission >>> fault. The question is why the address is in that range? The address should >>> be somewhere in GOT in the 5th range - 0x0000100000041000 >>> 0x0000100000042000 [4.0 kB] flags=fmF perm=rw offset=0x00035000 >>> path=/libzfs.so which has read/write permission. >>> >>> On Ubuntu host when I run the same command like and add extra debugging >>> to print statements like this: >>> >>> ELF [tid:36, mod:5, /*libzfs*.so]: arch_relocate_jump_slot, >>> addr:0x10000007a8d0 >>> They all print addresses within the range 0x10000007a000 - >>> 0x10000007b000 which are read-write permitted as they should be: >>> 0x0000100000044000 0x000010000004e000 [40.0 kB] flags=fmF >>> perm=r offset=0x00000000 path=/libzfs.so >>> 0x000010000004e000 0x000010000006f000 [132.0 kB] flags=fmF >>> perm=rx offset=0x0000a000 path=/libzfs.so >>> 0x000010000006f000 0x0000100000079000 [40.0 kB] flags=fmF >>> perm=r offset=0x0002b000 path=/libzfs.so >>> 0x0000100000079000 0x000010000007a000 [4.0 kB] flags=fmF >>> perm=r offset=0x00034000 path=/libzfs.so >>> *0x000010000007a000 0x000010000007c000 [8.0 kB] flags=fmF >>> perm=rw offset=0x00035000 path=/libzfs.so * >>> >>> I wonder if we have a bug when calculating where each segment should be >>> mapped: >>> >>> 400 void file::load_segment(const Elf64_Phdr& phdr) >>> 401 { >>> 402 ulong vstart = align_down(phdr.p_vaddr, mmu::page_size); >>> 403 ulong filesz_unaligned = phdr.p_vaddr + phdr.p_filesz - vstart; >>> 404 ulong filesz = align_up(filesz_unaligned, mmu::page_size); >>> 405 ulong memsz = align_up(phdr.p_vaddr + phdr.p_memsz, >>> mmu::page_size) - vstart; >>> 406 >>> 407 unsigned perm = get_segment_mmap_permissions(phdr); >>> 408 >>> 409 auto flag = mmu::mmap_fixed | (mlocked() ? mmu::mmap_populate : >>> 0); >>> 410 mmu::map_file(_base + vstart, filesz, flag, perm, _f, >>> align_down(phdr.p_offset, mmu::page_size)); >>> 411 if (phdr.p_filesz != phdr.p_memsz) { >>> 412 assert(perm & mmu::perm_write); >>> 413 memset(_base + vstart + filesz_unaligned, 0, filesz - >>> filesz_unaligned); >>> 414 if (memsz != filesz) { >>> 415 mmu::map_anon(_base + vstart + filesz, memsz - filesz, >>> flag, perm); >>> 416 } >>> 417 } >>> 418 elf_debug("Loaded and mapped PT_LOAD segment at: %018p of size: >>> 0x%x\n", _base + vstart, filesz); >>> 419 } >>> >>> BTW I am also interested what the output of this readelf command for >>> your libzfs.so is. Mine is this: >>> >>> readelf -l build/release/libzfs-stripped.so >>> >>> Elf file type is DYN (Shared object file) >>> Entry point 0xd1d0 >>> There are 11 program headers, starting at offset 64 >>> >>> Program Headers: >>> Type Offset VirtAddr PhysAddr >>> FileSiz MemSiz Flags Align >>> LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000 >>> 0x00000000000098e0 0x00000000000098e0 R 0x1000 >>> LOAD 0x000000000000a000 0x000000000000a000 0x000000000000a000 >>> 0x00000000000201a1 0x00000000000201a1 R E 0x1000 >>> LOAD 0x000000000002b000 0x000000000002b000 0x000000000002b000 >>> 0x0000000000009258 0x0000000000009258 R 0x1000 >>> LOAD 0x0000000000034cb0 0x0000000000035cb0 0x0000000000035cb0 >>> 0x00000000000017f0 0x0000000000002c00 RW 0x1000 >>> DYNAMIC 0x0000000000034d80 0x0000000000035d80 0x0000000000035d80 >>> 0x00000000000001d0 0x00000000000001d0 RW 0x8 >>> NOTE 0x00000000000002a8 0x00000000000002a8 0x00000000000002a8 >>> 0x0000000000000020 0x0000000000000020 R 0x8 >>> NOTE 0x00000000000002c8 0x00000000000002c8 0x00000000000002c8 >>> 0x0000000000000024 0x0000000000000024 R 0x4 >>> GNU_PROPERTY 0x00000000000002a8 0x00000000000002a8 0x00000000000002a8 >>> 0x0000000000000020 0x0000000000000020 R 0x8 >>> GNU_EH_FRAME 0x000000000002f768 0x000000000002f768 0x000000000002f768 >>> 0x0000000000000cec 0x0000000000000cec R 0x4 >>> GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 >>> 0x0000000000000000 0x0000000000000000 RW 0x10 >>> GNU_RELRO 0x0000000000034cb0 0x0000000000035cb0 0x0000000000035cb0 >>> 0x0000000000000350 0x0000000000000350 R 0x1 >>> >>> Section to Segment mapping: >>> Segment Sections... >>> 00 .note.gnu.property .note.gnu.build-id .gnu.hash .dynsym >>> .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt >>> 01 .init .plt .plt.got .plt.sec .text .fini >>> 02 .rodata .eh_frame_hdr .eh_frame >>> 03 .init_array .fini_array .data.rel.ro .dynamic .got .got.plt >>> .data .bss >>> 04 .dynamic >>> 05 .note.gnu.property >>> 06 .note.gnu.build-id >>> 07 .note.gnu.property >>> 08 .eh_frame_hdr >>> 09 >>> 10 .init_array .fini_array .data.rel.ro .dynamic .got >>> >>> On Tuesday, December 8, 2020 at 11:39:43 AM UTC-5 Matthew Kenigsberg >>> wrote: >>> >>>> My gdb is not the strongest but if I hbreak on arch_relocate_jump_slot >>>> looks like _pathname is /libzfs.so, eventually /zpool.so, and then a >>>> single >>>> /libzfs.so before continue hangs >>>> >>>> On Tuesday, December 8, 2020 at 9:11:15 AM UTC-7 Matthew Kenigsberg >>>> wrote: >>>> >>>>> Nix is a package manager, and NixOS is an operating system built >>>>> completely around the package manager. So every library is stored >>>>> somewhere >>>>> in /nix/store, like for example on Nix there is never anything like >>>>> /lib64/ld-linux-x86-64.so. >>>>> It would be /nix/store/.../ld-linux-x86-64.so. I could install the >>>>> package >>>>> manager on a different OS, in which case I might have both /lib64 and >>>>> /nix/store, but on NixOS I'll just have the latter. Does that make sense? >>>>> Not >>>>> sure if that's messing up something with linking. Guessing I can't >>>>> reproduce the error on any other OS, but happy to try. >>>>> On Tuesday, December 8, 2020 at 9:05:11 AM UTC-7 Matthew Kenigsberg >>>>> wrote: >>>>> >>>>>> (gdb) connect >>>>>> abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: >>>>>> %d)\n") at runtime.cc:105 >>>>>> 105 do {} while (true); >>>>>> (gdb) osv syms >>>>>> manifest.find_file: path=/libvdso.so, found file=libvdso.so >>>>>> /home/matthew/osv/build/release.x64/libvdso.so 0x100000000000 >>>>>> add symbol table from file >>>>>> "/home/matthew/osv/build/release.x64/libvdso.so" at >>>>>> .text_addr = 0x100000001040 >>>>>> .hash_addr = 0x1000000001c8 >>>>>> .gnu.hash_addr = 0x100000000200 >>>>>> .dynsym_addr = 0x100000000238 >>>>>> .dynstr_addr = 0x1000000002f8 >>>>>> .gnu.version_addr = 0x1000000003be >>>>>> .gnu.version_d_addr = 0x1000000003d0 >>>>>> .rela.plt_addr = 0x100000000408 >>>>>> .plt_addr = 0x100000001000 >>>>>> .eh_frame_addr = 0x100000002000 >>>>>> .dynamic_addr = 0x100000003e60 >>>>>> .got_addr = 0x100000003fd0 >>>>>> .comment_addr = 0x100000000000 >>>>>> .debug_aranges_addr = 0x100000000000 >>>>>> .debug_info_addr = 0x100000000000 >>>>>> .debug_abbrev_addr = 0x100000000000 >>>>>> .debug_line_addr = 0x100000000000 >>>>>> .debug_str_addr = 0x100000000000 >>>>>> .debug_loc_addr = 0x100000000000 >>>>>> .symtab_addr = 0x100000000000 >>>>>> .strtab_addr = 0x100000000000 >>>>>> warning: section .comment not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .debug_aranges not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .debug_info not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .debug_abbrev not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .debug_line not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .debug_str not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .debug_loc not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .symtab not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> warning: section .strtab not found in >>>>>> /home/matthew/osv/build/release.x64/libvdso.so >>>>>> manifest.find_file: path=/tools/mkfs.so, found file=tools/mkfs/mkfs.so >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so 0x100000004000 >>>>>> add symbol table from file >>>>>> "/home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so" at >>>>>> .text_addr = 0x100000006250 >>>>>> .hash_addr = 0x100000004200 >>>>>> .gnu.hash_addr = 0x100000004360 >>>>>> .dynsym_addr = 0x1000000043c0 >>>>>> .dynstr_addr = 0x100000004840 >>>>>> .gnu.version_addr = 0x100000005092 >>>>>> .gnu.version_r_addr = 0x1000000050f8 >>>>>> .rela.dyn_addr = 0x100000005148 >>>>>> .rela.plt_addr = 0x100000005298 >>>>>> .init_addr = 0x100000006000 >>>>>> .plt_addr = 0x100000006020 >>>>>> .plt.got_addr = 0x100000006240 >>>>>> .fini_addr = 0x10000000737c >>>>>> .rodata_addr = 0x100000008000 >>>>>> .eh_frame_hdr_addr = 0x10000000817c >>>>>> .eh_frame_addr = 0x100000008210 >>>>>> .gcc_except_table_addr = 0x100000008530 >>>>>> .init_array_addr = 0x100000009c60 >>>>>> .fini_array_addr = 0x100000009c70 >>>>>> .dynamic_addr = 0x100000009c78 >>>>>> .got_addr = 0x100000009e98 >>>>>> .data_addr = 0x10000000a000 >>>>>> .bss_addr = 0x10000000a010 >>>>>> .comment_addr = 0x100000004000 >>>>>> .debug_aranges_addr = 0x100000004000 >>>>>> .debug_info_addr = 0x100000004000 >>>>>> .debug_abbrev_addr = 0x100000004000 >>>>>> --Type <RET> for more, q to quit, c to continue without paging--c >>>>>> .debug_line_addr = 0x100000004000 >>>>>> .debug_str_addr = 0x100000004000 >>>>>> .debug_loc_addr = 0x100000004000 >>>>>> .debug_ranges_addr = 0x100000004000 >>>>>> .symtab_addr = 0x100000004000 >>>>>> .strtab_addr = 0x100000004000 >>>>>> warning: section .comment not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_aranges not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_info not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_abbrev not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_line not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_str not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_loc not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .debug_ranges not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .symtab not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> warning: section .strtab not found in >>>>>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so >>>>>> manifest.find_file: path=/libzfs.so, found file=libzfs.so >>>>>> /home/matthew/osv/build/release.x64/libzfs.so 0x10000000b000 >>>>>> add symbol table from file >>>>>> "/home/matthew/osv/build/release.x64/libzfs.so" at >>>>>> .text_addr = 0x1000000178f0 >>>>>> .hash_addr = 0x10000000b200 >>>>>> .gnu.hash_addr = 0x10000000c318 >>>>>> .dynsym_addr = 0x10000000cd78 >>>>>> .dynstr_addr = 0x100000010300 >>>>>> .gnu.version_addr = 0x1000000124e2 >>>>>> .gnu.version_r_addr = 0x100000012958 >>>>>> .rela.dyn_addr = 0x1000000129b8 >>>>>> .rela.plt_addr = 0x1000000134b0 >>>>>> .init_addr = 0x100000016000 >>>>>> .plt_addr = 0x100000016020 >>>>>> .plt.got_addr = 0x1000000178e0 >>>>>> .fini_addr = 0x1000000340a0 >>>>>> .rodata_addr = 0x100000035000 >>>>>> .eh_frame_hdr_addr = 0x100000039768 >>>>>> .eh_frame_addr = 0x10000003a470 >>>>>> .init_array_addr = 0x100000040010 >>>>>> .fini_array_addr = 0x100000040018 >>>>>> .data.rel.ro_addr = 0x100000040020 >>>>>> .dynamic_addr = 0x1000000400e0 >>>>>> .got_addr = 0x1000000402f0 >>>>>> .data_addr = 0x100000041000 >>>>>> .bss_addr = 0x100000041820 >>>>>> .comment_addr = 0x10000000b000 >>>>>> .debug_aranges_addr = 0x10000000b000 >>>>>> .debug_info_addr = 0x10000000b000 >>>>>> .debug_abbrev_addr = 0x10000000b000 >>>>>> .debug_line_addr = 0x10000000b000 >>>>>> .debug_str_addr = 0x10000000b000 >>>>>> .debug_loc_addr = 0x10000000b000 >>>>>> .debug_ranges_addr = 0x10000000b000 >>>>>> .symtab_addr = 0x10000000b000 >>>>>> .strtab_addr = 0x10000000b000 >>>>>> warning: section .comment not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_aranges not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_info not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_abbrev not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_line not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_str not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_loc not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .debug_ranges not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .symtab not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> warning: section .strtab not found in >>>>>> /home/matthew/osv/build/release.x64/libzfs.so >>>>>> manifest.find_file: path=/libuutil.so, found file=libuutil.so >>>>>> /home/matthew/osv/build/release.x64/libuutil.so 0x100000043000 >>>>>> add symbol table from file >>>>>> "/home/matthew/osv/build/release.x64/libuutil.so" at >>>>>> .text_addr = 0x1000000463c0 >>>>>> .hash_addr = 0x100000043200 >>>>>> .gnu.hash_addr = 0x100000043640 >>>>>> .dynsym_addr = 0x1000000438f8 >>>>>> .dynstr_addr = 0x100000044600 >>>>>> .gnu.version_addr = 0x100000044da4 >>>>>> .gnu.version_r_addr = 0x100000044ec0 >>>>>> .rela.dyn_addr = 0x100000044f00 >>>>>> .rela.plt_addr = 0x100000045068 >>>>>> .init_addr = 0x100000046000 >>>>>> .plt_addr = 0x100000046020 >>>>>> .plt.got_addr = 0x1000000463b0 >>>>>> .fini_addr = 0x100000049adc >>>>>> .rodata_addr = 0x10000004a000 >>>>>> .eh_frame_hdr_addr = 0x10000004ac84 >>>>>> .eh_frame_addr = 0x10000004af90 >>>>>> .init_array_addr = 0x10000004dbd8 >>>>>> .fini_array_addr = 0x10000004dbe0 >>>>>> .dynamic_addr = 0x10000004dbe8 >>>>>> .got_addr = 0x10000004dde8 >>>>>> .data_addr = 0x10000004e000 >>>>>> .bss_addr = 0x10000004e260 >>>>>> .comment_addr = 0x100000043000 >>>>>> .debug_aranges_addr = 0x100000043000 >>>>>> .debug_info_addr = 0x100000043000 >>>>>> .debug_abbrev_addr = 0x100000043000 >>>>>> .debug_line_addr = 0x100000043000 >>>>>> .debug_str_addr = 0x100000043000 >>>>>> .debug_loc_addr = 0x100000043000 >>>>>> .debug_ranges_addr = 0x100000043000 >>>>>> .symtab_addr = 0x100000043000 >>>>>> .strtab_addr = 0x100000043000 >>>>>> warning: section .comment not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_aranges not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_info not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_abbrev not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_line not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_str not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_loc not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .debug_ranges not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .symtab not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> warning: section .strtab not found in >>>>>> /home/matthew/osv/build/release.x64/libuutil.so >>>>>> manifest.find_file: path=/usr/lib/libgcc_s.so.1, found >>>>>> file=%(libgcc_s_dir)s/libgcc_s.so.1 >>>>>> ERROR: Unable to locate object file for: /usr/lib/libgcc_s.so.1 >>>>>> 0x10000004f000 >>>>>> manifest.find_file: path=/zpool.so, found file=zpool.so >>>>>> /home/matthew/osv/build/release.x64/zpool.so 0x100000069000 >>>>>> add symbol table from file >>>>>> "/home/matthew/osv/build/release.x64/zpool.so" at >>>>>> .text_addr = 0x10000006ebc0 >>>>>> .hash_addr = 0x100000069200 >>>>>> .gnu.hash_addr = 0x100000069868 >>>>>> .dynsym_addr = 0x100000069968 >>>>>> .dynstr_addr = 0x10000006ad18 >>>>>> .gnu.version_addr = 0x10000006b958 >>>>>> .gnu.version_r_addr = 0x10000006bb00 >>>>>> .rela.dyn_addr = 0x10000006bb30 >>>>>> .rela.plt_addr = 0x10000006c208 >>>>>> .init_addr = 0x10000006e000 >>>>>> .plt_addr = 0x10000006e020 >>>>>> .plt.got_addr = 0x10000006ebb0 >>>>>> .fini_addr = 0x10000007a420 >>>>>> .rodata_addr = 0x10000007b000 >>>>>> .eh_frame_hdr_addr = 0x10000007fd30 >>>>>> .eh_frame_addr = 0x100000080028 >>>>>> .init_array_addr = 0x100000082758 >>>>>> .fini_array_addr = 0x100000082760 >>>>>> .dynamic_addr = 0x100000082768 >>>>>> .got_addr = 0x100000082978 >>>>>> .data_addr = 0x100000083000 >>>>>> .bss_addr = 0x100000083360 >>>>>> .comment_addr = 0x100000069000 >>>>>> .debug_aranges_addr = 0x100000069000 >>>>>> .debug_info_addr = 0x100000069000 >>>>>> .debug_abbrev_addr = 0x100000069000 >>>>>> .debug_line_addr = 0x100000069000 >>>>>> .debug_str_addr = 0x100000069000 >>>>>> .debug_loc_addr = 0x100000069000 >>>>>> .debug_ranges_addr = 0x100000069000 >>>>>> .symtab_addr = 0x100000069000 >>>>>> .strtab_addr = 0x100000069000 >>>>>> warning: section .comment not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_aranges not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_info not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_abbrev not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_line not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_str not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_loc not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .debug_ranges not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .symtab not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> warning: section .strtab not found in >>>>>> /home/matthew/osv/build/release.x64/zpool.so >>>>>> (gdb) osv mmap >>>>>> 0x0000000000000000 0x0000000000000000 [0.0 kB] flags=none perm=none >>>>>> 0x0000100000000000 0x0000100000001000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00000000 path=/libvdso.so >>>>>> 0x0000100000001000 0x0000100000002000 [4.0 kB] flags=fmF perm=rx >>>>>> offset=0x00001000 path=/libvdso.so >>>>>> 0x0000100000002000 0x0000100000003000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00002000 path=/libvdso.so >>>>>> 0x0000100000003000 0x0000100000004000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00002000 path=/libvdso.so >>>>>> 0x0000100000004000 0x0000100000006000 [8.0 kB] flags=fmF perm=r >>>>>> offset=0x00000000 path=/tools/mkfs.so >>>>>> 0x0000100000006000 0x0000100000008000 [8.0 kB] flags=fmF perm=rx >>>>>> offset=0x00002000 path=/tools/mkfs.so >>>>>> 0x0000100000008000 0x0000100000009000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00004000 path=/tools/mkfs.so >>>>>> 0x0000100000009000 0x000010000000a000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00004000 path=/tools/mkfs.so >>>>>> 0x000010000000a000 0x000010000000b000 [4.0 kB] flags=fmF perm=rw >>>>>> offset=0x00005000 path=/tools/mkfs.so >>>>>> 0x000010000000b000 0x0000100000016000 [44.0 kB] flags=fmF perm=r >>>>>> offset=0x00000000 path=/libzfs.so >>>>>> 0x0000100000016000 0x0000100000035000 [124.0 kB] flags=fmF perm=rx >>>>>> offset=0x0000b000 path=/libzfs.so >>>>>> 0x0000100000035000 0x000010000003f000 [40.0 kB] flags=fmF perm=r >>>>>> offset=0x0002a000 path=/libzfs.so >>>>>> 0x0000100000040000 0x0000100000041000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00034000 path=/libzfs.so >>>>>> 0x0000100000041000 0x0000100000042000 [4.0 kB] flags=fmF perm=rw >>>>>> offset=0x00035000 path=/libzfs.so >>>>>> 0x0000100000042000 0x0000100000043000 [4.0 kB] flags=f perm=rw >>>>>> 0x0000100000043000 0x0000100000046000 [12.0 kB] flags=fmF perm=r >>>>>> offset=0x00000000 path=/libuutil.so >>>>>> 0x0000100000046000 0x000010000004a000 [16.0 kB] flags=fmF perm=rx >>>>>> offset=0x00003000 path=/libuutil.so >>>>>> 0x000010000004a000 0x000010000004c000 [8.0 kB] flags=fmF perm=r >>>>>> offset=0x00007000 path=/libuutil.so >>>>>> 0x000010000004d000 0x000010000004e000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00009000 path=/libuutil.so >>>>>> 0x000010000004e000 0x000010000004f000 [4.0 kB] flags=fmF perm=rw >>>>>> offset=0x0000a000 path=/libuutil.so >>>>>> 0x000010000004f000 0x0000100000052000 [12.0 kB] flags=fmF perm=r >>>>>> offset=0x00000000 path=/usr/lib/libgcc_s.so.1 >>>>>> 0x0000100000052000 0x0000100000063000 [68.0 kB] flags=fmF perm=rx >>>>>> offset=0x00003000 path=/usr/lib/libgcc_s.so.1 >>>>>> 0x0000100000063000 0x0000100000067000 [16.0 kB] flags=fmF perm=r >>>>>> offset=0x00014000 path=/usr/lib/libgcc_s.so.1 >>>>>> 0x0000100000067000 0x0000100000068000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00017000 path=/usr/lib/libgcc_s.so.1 >>>>>> 0x0000100000068000 0x0000100000069000 [4.0 kB] flags=fmF perm=rw >>>>>> offset=0x00018000 path=/usr/lib/libgcc_s.so.1 >>>>>> 0x0000100000069000 0x000010000006e000 [20.0 kB] flags=fmF perm=r >>>>>> offset=0x00000000 path=/zpool.so >>>>>> 0x000010000006e000 0x000010000007b000 [52.0 kB] flags=fmF perm=rx >>>>>> offset=0x00005000 path=/zpool.so >>>>>> 0x000010000007b000 0x0000100000082000 [28.0 kB] flags=fmF perm=r >>>>>> offset=0x00012000 path=/zpool.so >>>>>> 0x0000100000082000 0x0000100000083000 [4.0 kB] flags=fmF perm=r >>>>>> offset=0x00018000 path=/zpool.so >>>>>> 0x0000100000083000 0x0000100000084000 [4.0 kB] flags=fmF perm=rw >>>>>> offset=0x00019000 path=/zpool.so >>>>>> 0x0000100000084000 0x0000100000086000 [8.0 kB] flags=f perm=rw >>>>>> 0x0000200000000000 0x0000200000001000 [4.0 kB] flags=p perm=none >>>>>> 0x0000200000001000 0x0000200000002000 [4.0 kB] flags=p perm=none >>>>>> 0x0000200000002000 0x0000200000101000 [1020.0 kB] flags=p perm=rw >>>>>> 0x0000200000101000 0x0000200000102000 [4.0 kB] flags=p perm=none >>>>>> 0x0000200000102000 0x0000200000201000 [1020.0 kB] flags=p perm=rw >>>>>> 0x0000800000000000 0x0000800000000000 [0.0 kB] flags=none perm=none >>>>>> (gdb) bt >>>>>> #0 abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: >>>>>> %d)\n") at runtime.cc:105 >>>>>> #1 0x000000004023c75b in __assert_fail (expr=expr@entry=0x406742f8 >>>>>> "ef->rflags & processor::rflags_if", file=file@entry=0x40674325 >>>>>> "arch/x64/mmu.cc", line=line@entry=38, func=func@entry=0x4067431a >>>>>> "page_fault") at runtime.cc:139 >>>>>> #2 0x0000000040399305 in page_fault (ef=0xffff800000015048) at >>>>>> arch/x64/arch-cpu.hh:107 >>>>>> #3 <signal handler called> >>>>>> #4 0x000000004035ca19 in elf::object::symtab_len >>>>>> (this=0xffffa00000f19a00) at core/elf.cc:983 >>>>>> #5 0x000000004035cad8 in elf::object::lookup_addr >>>>>> (this=0xffffa00000f19a00, addr=addr@entry=0x1000000254ce >>>>>> <is_mounted+30>) >>>>>> at core/elf.cc:1015 >>>>>> #6 0x000000004035cca7 in elf::program::<lambda(const >>>>>> elf::program::modules_list&)>::operator() (__closure=<synthetic >>>>>> pointer>, >>>>>> __closure=<synthetic pointer>, ml=...) at core/elf.cc:1620 >>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void >>>>>> const*)::<lambda(const elf::program::modules_list&)> > (f=..., >>>>>> this=0xffffa00000097e70) at include/osv/elf.hh:702 >>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70, >>>>>> addr=addr@entry=0x1000000254ce <is_mounted+30>) at core/elf.cc:1617 >>>>>> #9 0x00000000404367cc in osv::lookup_name_demangled >>>>>> (addr=addr@entry=0x1000000254ce <is_mounted+30>, >>>>>> buf=buf@entry=0xffff8000012156d0 "???+19929103", len=len@entry=1024) at >>>>>> core/demangle.cc:47 >>>>>> #10 0x000000004023c540 in print_backtrace () at runtime.cc:85 >>>>>> #11 0x000000004023c714 in abort (fmt=fmt@entry=0x40645aff >>>>>> "Aborted\n") at runtime.cc:121 >>>>>> #12 0x0000000040202989 in abort () at runtime.cc:98 >>>>>> #13 0x0000000040345934 in mmu::vm_sigsegv (ef=0xffff800001216068, >>>>>> addr=<optimized out>) at core/mmu.cc:1314 >>>>>> #14 mmu::vm_sigsegv (addr=<optimized out>, ef=0xffff800001216068) at >>>>>> core/mmu.cc:1308 >>>>>> #15 0x000000004034782f in mmu::vm_fault >>>>>> (addr=addr@entry=17592186309800, ef=ef@entry=0xffff800001216068) at >>>>>> core/mmu.cc:1328 >>>>>> #16 0x00000000403992a3 in page_fault (ef=0xffff800001216068) at >>>>>> arch/x64/mmu.cc:42 >>>>>> #17 <signal handler called> >>>>>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot >>>>>> (this=this@entry=0xffffa00000f19a00, sym=..., >>>>>> addr=addr@entry=0x100000040ca8 <[email protected]>, >>>>>> addend=addend@entry=0) >>>>>> at arch/x64/arch-elf.cc:172 >>>>>> #19 0x0000000040361004 in elf::object::resolve_pltgot >>>>>> (this=0xffffa00000f19a00, index=<optimized out>) at core/elf.cc:843 >>>>>> #20 0x0000000040361229 in elf_resolve_pltgot (index=308, >>>>>> obj=0xffffa00000f19a00) at core/elf.cc:1860 >>>>>> #21 0x0000000040397d50 in __elf_resolve_pltgot () at >>>>>> arch/x64/elf-dl.S:47 >>>>>> #22 0x00001000000254cf in is_mounted (zfs_hdl=0x134, >>>>>> special=<optimized out>, where=0x403f3377 <malloc(size_t)+71>) at >>>>>> bsd/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_mount.c:214 >>>>>> #23 0xffff900000a99000 in ?? () >>>>>> #24 0x0000000000000000 in ?? () >>>>>> >>>>>> On Tuesday, December 8, 2020 at 8:53:52 AM UTC-7 [email protected] >>>>>> wrote: >>>>>> >>>>>>> It would be also nice to understand if we are crashing on the 1st >>>>>>> arch_relocate_jump_slot() for libfzs.so or is it a specific JUMP_SLOT >>>>>>> that >>>>>>> causes this crash? >>>>>>> >>>>>>> On Tuesday, December 8, 2020 at 10:39:06 AM UTC-5 Waldek Kozaczuk >>>>>>> wrote: >>>>>>> >>>>>>>> After you connect with gdb can you run 'osv mmap' and send us the >>>>>>>> output. Make sure you run 'osv syms' before it and dump backtrace >>>>>>>> after. >>>>>>>> Please see >>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv for any >>>>>>>> details. >>>>>>>> >>>>>>>> BTW can you build and run OSv ZFS image on the host without NIX? As >>>>>>>> I understand NIX is really just a layer on top of any Linux >>>>>>>> distribution, >>>>>>>> no? I am afraid I do not still understand what exactly NiX is I guess. >>>>>>>> >>>>>>>> >>>>>>>> On Monday, December 7, 2020 at 2:58:40 PM UTC-5 Matthew Kenigsberg >>>>>>>> wrote: >>>>>>>> >>>>>>>>> (gdb) frame 18 >>>>>>>>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot >>>>>>>>> (this=this@entry=0xffffa0000110fa00, sym=..., >>>>>>>>> addr=addr@entry=0x100000040ca8, addend=addend@entry=0) at >>>>>>>>> arch/x64/arch-elf.cc:172 >>>>>>>>> 172 *static_cast<void**>(addr) = sym.relocated_addr(); >>>>>>>>> (gdb) print _pathname >>>>>>>>> $14 = {static npos = 18446744073709551615, >>>>>>>>> _M_dataplus = {<std::allocator<char>> = >>>>>>>>> {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data >>>>>>>>> fields>}, >>>>>>>>> _M_p = 0xffffa0000110fa30 "/libzfs.so"}, _M_string_length = >>>>>>>>> 10, { >>>>>>>>> _M_local_buf = "/libzfs.so\000\000\000\000\000", >>>>>>>>> _M_allocated_capacity = 3347131623889529903}} >>>>>>>>> >>>>>>>>> Also been wondering if nix using nonstandard paths is causing >>>>>>>>> problems, like for libc: >>>>>>>>> [nix-shell:~/osv/build/release]$ ldd libzfs.so >>>>>>>>> linux-vdso.so.1 (0x00007ffcedbb9000) >>>>>>>>> libuutil.so => not found >>>>>>>>> libc.so.6 => >>>>>>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libc.so.6 >>>>>>>>> (0x00007f7594f38000) >>>>>>>>> >>>>>>>>> >>>>>>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib64/ld-linux-x86-64.so.2 >>>>>>>>> >>>>>>>>> (0x00007f7595131000) >>>>>>>>> On Sunday, December 6, 2020 at 8:43:10 AM UTC-7 [email protected] >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It might be easier to simply print '_pathname' value if you >>>>>>>>>> switch to the right frame in gdb. It would be nice to confirm that >>>>>>>>>> the >>>>>>>>>> problem we have is with zpool.so and that might lead to >>>>>>>>>> understanding why >>>>>>>>>> this crash happens. Maybe the is something wrong with building >>>>>>>>>> zpool.so. >>>>>>>>>> >>>>>>>>>> BTW based on this fragment of the stacktrace: >>>>>>>>>> >>>>>>>>>> #6 0x000000004035cb07 in elf::program::<lambda(const >>>>>>>>>> elf::program::modules_list&)>::operator() ( >>>>>>>>>> __closure=<synthetic pointer>, __closure=<synthetic pointer>, >>>>>>>>>> ml=...) at core/elf.cc:1620 >>>>>>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void >>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> > >>>>>>>>>> (f=..., this=0xffffa00000097e70) at include/osv/elf.hh:702 >>>>>>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70, >>>>>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617 >>>>>>>>>> #9 0x00000000404357cc in osv::lookup_name_demangled >>>>>>>>>> (addr=addr@entry=0x1000000254ce, >>>>>>>>>> buf=buf@entry=0xffff8000012146d0 "???+19630095", >>>>>>>>>> len=len@entry=1024) at core/demangle.cc:47 >>>>>>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85 >>>>>>>>>> >>>>>>>>>> It seems we have a bug (or need of improvement) in >>>>>>>>>> print_backtrace() to make it NOT try to demangle names like >>>>>>>>>> "???+19630095" >>>>>>>>>> which causes follow-up fault. >>>>>>>>>> >>>>>>>>>> At the same time, it is strange that we crash at line 983 which >>>>>>>>>> seems to indicate something goes wrong when processing zpool.so. >>>>>>>>>> >>>>>>>>>> 981 if (dynamic_exists(DT_HASH)) { >>>>>>>>>> >>>>>>>>>> 982 auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH); >>>>>>>>>> >>>>>>>>>> *983 return hashtab[1];* >>>>>>>>>> >>>>>>>>>> 984 } >>>>>>>>>> >>>>>>>>>> On Sunday, December 6, 2020 at 10:06:21 AM UTC-5 Waldek Kozaczuk >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Can you run the ROFS image you built? Also as I understand it >>>>>>>>>>> NIX is a package manager but what Linux distribution are you using? >>>>>>>>>>> >>>>>>>>>>> As far as ZFS goes could you enable ELF debugging - change this >>>>>>>>>>> line: >>>>>>>>>>> >>>>>>>>>>> conf-debug_elf=0 >>>>>>>>>>> >>>>>>>>>>> To >>>>>>>>>>> >>>>>>>>>>> conf-debug_elf=1 >>>>>>>>>>> >>>>>>>>>>> In conf/base.mk, delete core/elf.o and force rebuild the >>>>>>>>>>> kernel. I think you may also need to change the script >>>>>>>>>>> upload_manifest.py >>>>>>>>>>> to peeped ‘—verbose’ to the command line with cpiod.so >>>>>>>>>>> >>>>>>>>>>> It should show more info about elf loading. It may still be >>>>>>>>>>> necessary to add extra printouts to capture which exact elf it is >>>>>>>>>>> crashing >>>>>>>>>>> on in arch_relocate_jump(). >>>>>>>>>>> >>>>>>>>>>> In worst case I would need a copy of your loader-stripped.elf >>>>>>>>>>> and possibly all the other files like cpiod.so, zfs.so that go into >>>>>>>>>>> the >>>>>>>>>>> bootfs part of the image. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Waldek >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Dec 5, 2020 at 19:31 Matthew Kenigsberg < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> After forcing it to use the right path for libz.so.1, it's >>>>>>>>>>>> working with rofs, but still having the same issue when using zfs, >>>>>>>>>>>> even >>>>>>>>>>>> after I correct the path for libz. >>>>>>>>>>>> >>>>>>>>>>>> On Saturday, December 5, 2020 at 5:18:37 PM UTC-7 Matthew >>>>>>>>>>>> Kenigsberg wrote: >>>>>>>>>>>> >>>>>>>>>>>>> gcc version 9.3.0 (GCC) >>>>>>>>>>>>> QEMU emulator version 5.1.0 >>>>>>>>>>>>> >>>>>>>>>>>>> Running with fs=rofs I get the error: >>>>>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 369, >>>>>>>>>>>>> in <module> >>>>>>>>>>>>> main() >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 366, >>>>>>>>>>>>> in main >>>>>>>>>>>>> gen_image(outfile, manifest) >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 269, >>>>>>>>>>>>> in gen_image >>>>>>>>>>>>> system_structure_block, bytes_written = write_fs(fp, >>>>>>>>>>>>> manifest) >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 246, >>>>>>>>>>>>> in write_fs >>>>>>>>>>>>> count, directory_entries_index = write_dir(fp, >>>>>>>>>>>>> manifest.get(''), '', manifest) >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 207, >>>>>>>>>>>>> in write_dir >>>>>>>>>>>>> count, directory_entries_index = write_dir(fp, val, >>>>>>>>>>>>> dirpath + '/' + entry, manifest) >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 207, >>>>>>>>>>>>> in write_dir >>>>>>>>>>>>> count, directory_entries_index = write_dir(fp, val, >>>>>>>>>>>>> dirpath + '/' + entry, manifest) >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 222, >>>>>>>>>>>>> in write_dir >>>>>>>>>>>>> inode.count = write_file(fp, val) >>>>>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 164, >>>>>>>>>>>>> in write_file >>>>>>>>>>>>> with open(path, 'rb') as f: >>>>>>>>>>>>> FileNotFoundError: [Errno 2] No such file or directory: >>>>>>>>>>>>> 'libz.so.1' >>>>>>>>>>>>> >>>>>>>>>>>>> I think that's from this line in usr.manifest? >>>>>>>>>>>>> /usr/lib/libz.so.1: libz.so.1 >>>>>>>>>>>>> >>>>>>>>>>>>> Don't have zlib in the manifest without fs=rofs, and I think >>>>>>>>>>>>> zpool uses it? >>>>>>>>>>>>> >>>>>>>>>>>>> Looking into it... >>>>>>>>>>>>> On Saturday, December 5, 2020 at 4:36:20 PM UTC-7 >>>>>>>>>>>>> [email protected] wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I can not reproduce it on Ubuntu 20.20 neither Fedora 33. >>>>>>>>>>>>>> Here is the code fragment where it happens: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 169 bool object::arch_relocate_jump_slot(symbol_module& sym, >>>>>>>>>>>>>> void *addr, Elf64_Sxword addend) >>>>>>>>>>>>>> >>>>>>>>>>>>>> 170 { >>>>>>>>>>>>>> >>>>>>>>>>>>>> 171 if (sym.symbol) { >>>>>>>>>>>>>> >>>>>>>>>>>>>> 172 *static_cast<void**>(addr) = sym.relocated_addr(); >>>>>>>>>>>>>> >>>>>>>>>>>>>> 173 return true; >>>>>>>>>>>>>> >>>>>>>>>>>>>> 174 } else { >>>>>>>>>>>>>> >>>>>>>>>>>>>> 175 return false; >>>>>>>>>>>>>> >>>>>>>>>>>>>> 176 } >>>>>>>>>>>>>> >>>>>>>>>>>>>> 177 } >>>>>>>>>>>>>> It looks like writing at the addr 0x100000040ca8 in line 172 >>>>>>>>>>>>>> caused the fault. Why? >>>>>>>>>>>>>> >>>>>>>>>>>>>> And then the 2nd page fault in the gdb backtrace as the 1st >>>>>>>>>>>>>> one was being handled (not sure if that is a bug or just a state >>>>>>>>>>>>>> of loading >>>>>>>>>>>>>> of a program). >>>>>>>>>>>>>> >>>>>>>>>>>>>> 981 if (dynamic_exists(DT_HASH)) { >>>>>>>>>>>>>> >>>>>>>>>>>>>> 982 auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH); >>>>>>>>>>>>>> >>>>>>>>>>>>>> 983 return hashtab[1]; >>>>>>>>>>>>>> >>>>>>>>>>>>>> 984 } >>>>>>>>>>>>>> Is something wrong with the elf files cpiod.so, mkfs.so or >>>>>>>>>>>>>> zfs.so or something? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you try to do the same with ROFS? >>>>>>>>>>>>>> >>>>>>>>>>>>>> fs=rofs >>>>>>>>>>>>>> On Saturday, December 5, 2020 at 5:44:12 PM UTC-5 Matthew >>>>>>>>>>>>>> Kenigsberg wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Struggling to get scripts/build to run on NixOS because I'm >>>>>>>>>>>>>>> getting a page fault. NixOS does keep shared libraries in >>>>>>>>>>>>>>> nonstandard >>>>>>>>>>>>>>> locations, not sure if that's breaking something. More details >>>>>>>>>>>>>>> below, but >>>>>>>>>>>>>>> any ideas? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As far as I can tell, the error is caused by >>>>>>>>>>>>>>> tools/mkfs/mkfs.cc:71: >>>>>>>>>>>>>>> run_cmd("/zpool.so", zpool_args); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The error from scripts/build: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> OSv v0.55.0-145-g97f17a7a >>>>>>>>>>>>>>> eth0: 192.168.122.15 >>>>>>>>>>>>>>> Booted up in 154.38 ms >>>>>>>>>>>>>>> Cmdline: /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/zfs/; >>>>>>>>>>>>>>> /zfs.so set compression=off osv >>>>>>>>>>>>>>> Running mkfs... >>>>>>>>>>>>>>> page fault outside application, addr: 0x0000100000040ca8 >>>>>>>>>>>>>>> [registers] >>>>>>>>>>>>>>> RIP: 0x000000004039c25a >>>>>>>>>>>>>>> <elf::object::arch_relocate_jump_slot(elf::symbol_module&, >>>>>>>>>>>>>>> void*, long)+26> >>>>>>>>>>>>>>> RFL: 0x0000000000010202 CS: 0x0000000000000008 SS: >>>>>>>>>>>>>>> 0x0000000000000010 >>>>>>>>>>>>>>> RAX: 0x000010000007a340 RBX: 0x0000100000040ca8 RCX: >>>>>>>>>>>>>>> 0x000010000006abb0 RDX: 0x0000000000000002 >>>>>>>>>>>>>>> RSI: 0x00002000001f6f70 RDI: 0xffffa00001058c00 RBP: >>>>>>>>>>>>>>> 0x00002000001f6f30 R8: 0xffffa00000a68460 >>>>>>>>>>>>>>> R9: 0xffffa00000f18da0 R10: 0x0000000000000000 R11: >>>>>>>>>>>>>>> 0x00000000409dd380 R12: 0xffffa00000f18c00 >>>>>>>>>>>>>>> R13: 0xffffa00000f18da0 R14: 0x0000000000000000 R15: >>>>>>>>>>>>>>> 0x00000000409dd380 RSP: 0x00002000001f6f20 >>>>>>>>>>>>>>> Aborted >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [backtrace] >>>>>>>>>>>>>>> 0x00000000403458d3 <???+1077172435> >>>>>>>>>>>>>>> 0x00000000403477ce <mmu::vm_fault(unsigned long, >>>>>>>>>>>>>>> exception_frame*)+350> >>>>>>>>>>>>>>> 0x0000000040398ba2 <page_fault+162> >>>>>>>>>>>>>>> 0x0000000040397a16 <???+1077508630> >>>>>>>>>>>>>>> 0x0000000040360a13 <elf::object::resolve_pltgot(unsigned >>>>>>>>>>>>>>> int)+387> >>>>>>>>>>>>>>> 0x0000000040360c38 <elf_resolve_pltgot+56> >>>>>>>>>>>>>>> 0x000000004039764f <???+1077507663> >>>>>>>>>>>>>>> 0xffffa000012b880f <???+19630095> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trying to get a backtrace after connecting with gdb: >>>>>>>>>>>>>>> (gdb) bt >>>>>>>>>>>>>>> #0 abort (fmt=fmt@entry=0x40644b90 "Assertion failed: %s >>>>>>>>>>>>>>> (%s: %s: %d)\n") at runtime.cc:105 >>>>>>>>>>>>>>> #1 0x000000004023c6fb in __assert_fail >>>>>>>>>>>>>>> (expr=expr@entry=0x40672cf8 "ef->rflags & >>>>>>>>>>>>>>> processor::rflags_if", >>>>>>>>>>>>>>> file=file@entry=0x40672d25 "arch/x64/mmu.cc", >>>>>>>>>>>>>>> line=line@entry=38, func=func@entry=0x40672d1a "page_fault") >>>>>>>>>>>>>>> at runtime.cc:139 >>>>>>>>>>>>>>> #2 0x0000000040398c05 in page_fault (ef=0xffff800000015048) >>>>>>>>>>>>>>> at arch/x64/arch-cpu.hh:107 >>>>>>>>>>>>>>> #3 <signal handler called> >>>>>>>>>>>>>>> #4 0x000000004035c879 in elf::object::symtab_len >>>>>>>>>>>>>>> (this=0xffffa00000f18c00) at core/elf.cc:983 >>>>>>>>>>>>>>> #5 0x000000004035c938 in elf::object::lookup_addr >>>>>>>>>>>>>>> (this=0xffffa00000f18c00, addr=addr@entry=0x1000000254ce) >>>>>>>>>>>>>>> at core/elf.cc:1015 >>>>>>>>>>>>>>> #6 0x000000004035cb07 in elf::program::<lambda(const >>>>>>>>>>>>>>> elf::program::modules_list&)>::operator() ( >>>>>>>>>>>>>>> __closure=<synthetic pointer>, __closure=<synthetic >>>>>>>>>>>>>>> pointer>, ml=...) at core/elf.cc:1620 >>>>>>>>>>>>>>> #7 >>>>>>>>>>>>>>> elf::program::with_modules<elf::program::lookup_addr(void >>>>>>>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> > >>>>>>>>>>>>>>> (f=..., this=0xffffa00000097e70) at >>>>>>>>>>>>>>> include/osv/elf.hh:702 >>>>>>>>>>>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70, >>>>>>>>>>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617 >>>>>>>>>>>>>>> #9 0x00000000404357cc in osv::lookup_name_demangled >>>>>>>>>>>>>>> (addr=addr@entry=0x1000000254ce, >>>>>>>>>>>>>>> buf=buf@entry=0xffff8000012146d0 "???+19630095", >>>>>>>>>>>>>>> len=len@entry=1024) at core/demangle.cc:47 >>>>>>>>>>>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85 >>>>>>>>>>>>>>> #11 0x000000004023c6b4 in abort (fmt=fmt@entry=0x40644a9f >>>>>>>>>>>>>>> "Aborted\n") at runtime.cc:121 >>>>>>>>>>>>>>> #12 0x0000000040202989 in abort () at runtime.cc:98 >>>>>>>>>>>>>>> #13 0x00000000403458d4 in mmu::vm_sigsegv >>>>>>>>>>>>>>> (ef=0xffff800001215068, addr=<optimized out>) at >>>>>>>>>>>>>>> core/mmu.cc:1314 >>>>>>>>>>>>>>> #14 mmu::vm_sigsegv (addr=<optimized out>, >>>>>>>>>>>>>>> ef=0xffff800001215068) at core/mmu.cc:1308 >>>>>>>>>>>>>>> #15 0x00000000403477cf in mmu::vm_fault >>>>>>>>>>>>>>> (addr=addr@entry=17592186309800, ef=ef@entry=0xffff800001215068) >>>>>>>>>>>>>>> at core/mmu.cc:1328 >>>>>>>>>>>>>>> #16 0x0000000040398ba3 in page_fault (ef=0xffff800001215068) >>>>>>>>>>>>>>> at arch/x64/mmu.cc:42 >>>>>>>>>>>>>>> #17 <signal handler called> >>>>>>>>>>>>>>> #18 0x000000004039c25a in >>>>>>>>>>>>>>> elf::object::arch_relocate_jump_slot >>>>>>>>>>>>>>> (this=this@entry=0xffffa00000f18c00, >>>>>>>>>>>>>>> sym=..., >>>>>>>>>>>>>>> addr=addr@entry=0x100000040ca8, addend=addend@entry=0) >>>>>>>>>>>>>>> at arch/x64/arch-elf.cc:172 >>>>>>>>>>>>>>> #19 0x0000000040360a14 in elf::object::resolve_pltgot >>>>>>>>>>>>>>> (this=0xffffa00000f18c00, index=<optimized out>) >>>>>>>>>>>>>>> at core/elf.cc:843 >>>>>>>>>>>>>>> #20 0x0000000040360c39 in elf_resolve_pltgot (index=308, >>>>>>>>>>>>>>> obj=0xffffa00000f18c00) at core/elf.cc:1860 >>>>>>>>>>>>>>> #21 0x0000000040397650 in __elf_resolve_pltgot () at >>>>>>>>>>>>>>> arch/x64/elf-dl.S:47 >>>>>>>>>>>>>>> #22 0x00001000000254cf in ?? () >>>>>>>>>>>>>>> #23 0xffffa000012b8800 in ?? () >>>>>>>>>>>>>>> #24 0x00002000001f74a0 in ?? () >>>>>>>>>>>>>>> #25 0x00001000000254cf in ?? () >>>>>>>>>>>>>>> #26 0x00002000001f7480 in ?? () >>>>>>>>>>>>>>> #27 0x00000000403f241c in calloc (nmemb=<optimized out>, >>>>>>>>>>>>>>> size=<optimized out>) at core/mempool.cc:1811 >>>>>>>>>>>>>>> #28 0xffff900000a98000 in ?? () >>>>>>>>>>>>>>> #29 0x0000000000000000 in ?? () >>>>>>>>>>>>>>> On Saturday, November 28, 2020 at 1:39:46 PM UTC-7 Matthew >>>>>>>>>>>>>>> Kenigsberg wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'll send something, might take a bit before I find time to >>>>>>>>>>>>>>>> work on it though. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Matthew >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Saturday, November 28, 2020 at 1:11:11 PM UTC-7 Roman >>>>>>>>>>>>>>>> Shaposhnik wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Nov 24, 2020 at 8:03 AM Waldek Kozaczuk < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Hey, >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Send a patch with a new app that could demonstrate it, >>>>>>>>>>>>>>>>> please, if you can. I would like to see it. Sounds like a >>>>>>>>>>>>>>>>> nice improvement. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> FWIW: I'd love to see it too -- been meaning to play with >>>>>>>>>>>>>>>>> Nix and this >>>>>>>>>>>>>>>>> gives me a perfect excuse ;-) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Roman. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "OSv Development" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/osv-dev/rhjHPr7OBEw/unsubscribe >>>>>>>>>>>> . >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com >>>>>>>>>>>> >>>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/ea584c29-21cd-45a2-8905-993476117417n%40googlegroups.com.
