Nix is a package manager, and NixOS is an operating system built completely
around the package manager. So every library is stored somewhere in
/nix/store, like for example on Nix there is never anything like
/lib64/ld-linux-x86-64.so.
It would be /nix/store/.../ld-linux-x86-64.so. I could install the package
manager on a different OS, in which case I might have both /lib64 and
/nix/store, but on NixOS I'll just have the latter. Does that make sense? Not
sure if that's messing up something with linking. Guessing I can't
reproduce the error on any other OS, but happy to try.
On Tuesday, December 8, 2020 at 9:05:11 AM UTC-7 Matthew Kenigsberg wrote:
> (gdb) connect
> abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: %d)\n") at
> runtime.cc:105
> 105 do {} while (true);
> (gdb) osv syms
> manifest.find_file: path=/libvdso.so, found file=libvdso.so
> /home/matthew/osv/build/release.x64/libvdso.so 0x100000000000
> add symbol table from file
> "/home/matthew/osv/build/release.x64/libvdso.so" at
> .text_addr = 0x100000001040
> .hash_addr = 0x1000000001c8
> .gnu.hash_addr = 0x100000000200
> .dynsym_addr = 0x100000000238
> .dynstr_addr = 0x1000000002f8
> .gnu.version_addr = 0x1000000003be
> .gnu.version_d_addr = 0x1000000003d0
> .rela.plt_addr = 0x100000000408
> .plt_addr = 0x100000001000
> .eh_frame_addr = 0x100000002000
> .dynamic_addr = 0x100000003e60
> .got_addr = 0x100000003fd0
> .comment_addr = 0x100000000000
> .debug_aranges_addr = 0x100000000000
> .debug_info_addr = 0x100000000000
> .debug_abbrev_addr = 0x100000000000
> .debug_line_addr = 0x100000000000
> .debug_str_addr = 0x100000000000
> .debug_loc_addr = 0x100000000000
> .symtab_addr = 0x100000000000
> .strtab_addr = 0x100000000000
> warning: section .comment not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .debug_aranges not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .debug_info not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .debug_abbrev not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .debug_line not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .debug_str not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .debug_loc not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .symtab not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> warning: section .strtab not found in
> /home/matthew/osv/build/release.x64/libvdso.so
> manifest.find_file: path=/tools/mkfs.so, found file=tools/mkfs/mkfs.so
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so 0x100000004000
> add symbol table from file
> "/home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so" at
> .text_addr = 0x100000006250
> .hash_addr = 0x100000004200
> .gnu.hash_addr = 0x100000004360
> .dynsym_addr = 0x1000000043c0
> .dynstr_addr = 0x100000004840
> .gnu.version_addr = 0x100000005092
> .gnu.version_r_addr = 0x1000000050f8
> .rela.dyn_addr = 0x100000005148
> .rela.plt_addr = 0x100000005298
> .init_addr = 0x100000006000
> .plt_addr = 0x100000006020
> .plt.got_addr = 0x100000006240
> .fini_addr = 0x10000000737c
> .rodata_addr = 0x100000008000
> .eh_frame_hdr_addr = 0x10000000817c
> .eh_frame_addr = 0x100000008210
> .gcc_except_table_addr = 0x100000008530
> .init_array_addr = 0x100000009c60
> .fini_array_addr = 0x100000009c70
> .dynamic_addr = 0x100000009c78
> .got_addr = 0x100000009e98
> .data_addr = 0x10000000a000
> .bss_addr = 0x10000000a010
> .comment_addr = 0x100000004000
> .debug_aranges_addr = 0x100000004000
> .debug_info_addr = 0x100000004000
> .debug_abbrev_addr = 0x100000004000
> --Type <RET> for more, q to quit, c to continue without paging--c
> .debug_line_addr = 0x100000004000
> .debug_str_addr = 0x100000004000
> .debug_loc_addr = 0x100000004000
> .debug_ranges_addr = 0x100000004000
> .symtab_addr = 0x100000004000
> .strtab_addr = 0x100000004000
> warning: section .comment not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_aranges not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_info not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_abbrev not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_line not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_str not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_loc not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .debug_ranges not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .symtab not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> warning: section .strtab not found in
> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
> manifest.find_file: path=/libzfs.so, found file=libzfs.so
> /home/matthew/osv/build/release.x64/libzfs.so 0x10000000b000
> add symbol table from file "/home/matthew/osv/build/release.x64/libzfs.so"
> at
> .text_addr = 0x1000000178f0
> .hash_addr = 0x10000000b200
> .gnu.hash_addr = 0x10000000c318
> .dynsym_addr = 0x10000000cd78
> .dynstr_addr = 0x100000010300
> .gnu.version_addr = 0x1000000124e2
> .gnu.version_r_addr = 0x100000012958
> .rela.dyn_addr = 0x1000000129b8
> .rela.plt_addr = 0x1000000134b0
> .init_addr = 0x100000016000
> .plt_addr = 0x100000016020
> .plt.got_addr = 0x1000000178e0
> .fini_addr = 0x1000000340a0
> .rodata_addr = 0x100000035000
> .eh_frame_hdr_addr = 0x100000039768
> .eh_frame_addr = 0x10000003a470
> .init_array_addr = 0x100000040010
> .fini_array_addr = 0x100000040018
> .data.rel.ro_addr = 0x100000040020
> .dynamic_addr = 0x1000000400e0
> .got_addr = 0x1000000402f0
> .data_addr = 0x100000041000
> .bss_addr = 0x100000041820
> .comment_addr = 0x10000000b000
> .debug_aranges_addr = 0x10000000b000
> .debug_info_addr = 0x10000000b000
> .debug_abbrev_addr = 0x10000000b000
> .debug_line_addr = 0x10000000b000
> .debug_str_addr = 0x10000000b000
> .debug_loc_addr = 0x10000000b000
> .debug_ranges_addr = 0x10000000b000
> .symtab_addr = 0x10000000b000
> .strtab_addr = 0x10000000b000
> warning: section .comment not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_aranges not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_info not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_abbrev not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_line not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_str not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_loc not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .debug_ranges not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .symtab not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> warning: section .strtab not found in
> /home/matthew/osv/build/release.x64/libzfs.so
> manifest.find_file: path=/libuutil.so, found file=libuutil.so
> /home/matthew/osv/build/release.x64/libuutil.so 0x100000043000
> add symbol table from file
> "/home/matthew/osv/build/release.x64/libuutil.so" at
> .text_addr = 0x1000000463c0
> .hash_addr = 0x100000043200
> .gnu.hash_addr = 0x100000043640
> .dynsym_addr = 0x1000000438f8
> .dynstr_addr = 0x100000044600
> .gnu.version_addr = 0x100000044da4
> .gnu.version_r_addr = 0x100000044ec0
> .rela.dyn_addr = 0x100000044f00
> .rela.plt_addr = 0x100000045068
> .init_addr = 0x100000046000
> .plt_addr = 0x100000046020
> .plt.got_addr = 0x1000000463b0
> .fini_addr = 0x100000049adc
> .rodata_addr = 0x10000004a000
> .eh_frame_hdr_addr = 0x10000004ac84
> .eh_frame_addr = 0x10000004af90
> .init_array_addr = 0x10000004dbd8
> .fini_array_addr = 0x10000004dbe0
> .dynamic_addr = 0x10000004dbe8
> .got_addr = 0x10000004dde8
> .data_addr = 0x10000004e000
> .bss_addr = 0x10000004e260
> .comment_addr = 0x100000043000
> .debug_aranges_addr = 0x100000043000
> .debug_info_addr = 0x100000043000
> .debug_abbrev_addr = 0x100000043000
> .debug_line_addr = 0x100000043000
> .debug_str_addr = 0x100000043000
> .debug_loc_addr = 0x100000043000
> .debug_ranges_addr = 0x100000043000
> .symtab_addr = 0x100000043000
> .strtab_addr = 0x100000043000
> warning: section .comment not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_aranges not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_info not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_abbrev not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_line not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_str not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_loc not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .debug_ranges not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .symtab not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> warning: section .strtab not found in
> /home/matthew/osv/build/release.x64/libuutil.so
> manifest.find_file: path=/usr/lib/libgcc_s.so.1, found
> file=%(libgcc_s_dir)s/libgcc_s.so.1
> ERROR: Unable to locate object file for: /usr/lib/libgcc_s.so.1
> 0x10000004f000
> manifest.find_file: path=/zpool.so, found file=zpool.so
> /home/matthew/osv/build/release.x64/zpool.so 0x100000069000
> add symbol table from file "/home/matthew/osv/build/release.x64/zpool.so"
> at
> .text_addr = 0x10000006ebc0
> .hash_addr = 0x100000069200
> .gnu.hash_addr = 0x100000069868
> .dynsym_addr = 0x100000069968
> .dynstr_addr = 0x10000006ad18
> .gnu.version_addr = 0x10000006b958
> .gnu.version_r_addr = 0x10000006bb00
> .rela.dyn_addr = 0x10000006bb30
> .rela.plt_addr = 0x10000006c208
> .init_addr = 0x10000006e000
> .plt_addr = 0x10000006e020
> .plt.got_addr = 0x10000006ebb0
> .fini_addr = 0x10000007a420
> .rodata_addr = 0x10000007b000
> .eh_frame_hdr_addr = 0x10000007fd30
> .eh_frame_addr = 0x100000080028
> .init_array_addr = 0x100000082758
> .fini_array_addr = 0x100000082760
> .dynamic_addr = 0x100000082768
> .got_addr = 0x100000082978
> .data_addr = 0x100000083000
> .bss_addr = 0x100000083360
> .comment_addr = 0x100000069000
> .debug_aranges_addr = 0x100000069000
> .debug_info_addr = 0x100000069000
> .debug_abbrev_addr = 0x100000069000
> .debug_line_addr = 0x100000069000
> .debug_str_addr = 0x100000069000
> .debug_loc_addr = 0x100000069000
> .debug_ranges_addr = 0x100000069000
> .symtab_addr = 0x100000069000
> .strtab_addr = 0x100000069000
> warning: section .comment not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_aranges not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_info not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_abbrev not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_line not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_str not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_loc not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .debug_ranges not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .symtab not found in
> /home/matthew/osv/build/release.x64/zpool.so
> warning: section .strtab not found in
> /home/matthew/osv/build/release.x64/zpool.so
> (gdb) osv mmap
> 0x0000000000000000 0x0000000000000000 [0.0 kB] flags=none perm=none
> 0x0000100000000000 0x0000100000001000 [4.0 kB] flags=fmF perm=r
> offset=0x00000000 path=/libvdso.so
> 0x0000100000001000 0x0000100000002000 [4.0 kB] flags=fmF perm=rx
> offset=0x00001000 path=/libvdso.so
> 0x0000100000002000 0x0000100000003000 [4.0 kB] flags=fmF perm=r
> offset=0x00002000 path=/libvdso.so
> 0x0000100000003000 0x0000100000004000 [4.0 kB] flags=fmF perm=r
> offset=0x00002000 path=/libvdso.so
> 0x0000100000004000 0x0000100000006000 [8.0 kB] flags=fmF perm=r
> offset=0x00000000 path=/tools/mkfs.so
> 0x0000100000006000 0x0000100000008000 [8.0 kB] flags=fmF perm=rx
> offset=0x00002000 path=/tools/mkfs.so
> 0x0000100000008000 0x0000100000009000 [4.0 kB] flags=fmF perm=r
> offset=0x00004000 path=/tools/mkfs.so
> 0x0000100000009000 0x000010000000a000 [4.0 kB] flags=fmF perm=r
> offset=0x00004000 path=/tools/mkfs.so
> 0x000010000000a000 0x000010000000b000 [4.0 kB] flags=fmF perm=rw
> offset=0x00005000 path=/tools/mkfs.so
> 0x000010000000b000 0x0000100000016000 [44.0 kB] flags=fmF perm=r
> offset=0x00000000 path=/libzfs.so
> 0x0000100000016000 0x0000100000035000 [124.0 kB] flags=fmF perm=rx
> offset=0x0000b000 path=/libzfs.so
> 0x0000100000035000 0x000010000003f000 [40.0 kB] flags=fmF perm=r
> offset=0x0002a000 path=/libzfs.so
> 0x0000100000040000 0x0000100000041000 [4.0 kB] flags=fmF perm=r
> offset=0x00034000 path=/libzfs.so
> 0x0000100000041000 0x0000100000042000 [4.0 kB] flags=fmF perm=rw
> offset=0x00035000 path=/libzfs.so
> 0x0000100000042000 0x0000100000043000 [4.0 kB] flags=f perm=rw
> 0x0000100000043000 0x0000100000046000 [12.0 kB] flags=fmF perm=r
> offset=0x00000000 path=/libuutil.so
> 0x0000100000046000 0x000010000004a000 [16.0 kB] flags=fmF perm=rx
> offset=0x00003000 path=/libuutil.so
> 0x000010000004a000 0x000010000004c000 [8.0 kB] flags=fmF perm=r
> offset=0x00007000 path=/libuutil.so
> 0x000010000004d000 0x000010000004e000 [4.0 kB] flags=fmF perm=r
> offset=0x00009000 path=/libuutil.so
> 0x000010000004e000 0x000010000004f000 [4.0 kB] flags=fmF perm=rw
> offset=0x0000a000 path=/libuutil.so
> 0x000010000004f000 0x0000100000052000 [12.0 kB] flags=fmF perm=r
> offset=0x00000000 path=/usr/lib/libgcc_s.so.1
> 0x0000100000052000 0x0000100000063000 [68.0 kB] flags=fmF perm=rx
> offset=0x00003000 path=/usr/lib/libgcc_s.so.1
> 0x0000100000063000 0x0000100000067000 [16.0 kB] flags=fmF perm=r
> offset=0x00014000 path=/usr/lib/libgcc_s.so.1
> 0x0000100000067000 0x0000100000068000 [4.0 kB] flags=fmF perm=r
> offset=0x00017000 path=/usr/lib/libgcc_s.so.1
> 0x0000100000068000 0x0000100000069000 [4.0 kB] flags=fmF perm=rw
> offset=0x00018000 path=/usr/lib/libgcc_s.so.1
> 0x0000100000069000 0x000010000006e000 [20.0 kB] flags=fmF perm=r
> offset=0x00000000 path=/zpool.so
> 0x000010000006e000 0x000010000007b000 [52.0 kB] flags=fmF perm=rx
> offset=0x00005000 path=/zpool.so
> 0x000010000007b000 0x0000100000082000 [28.0 kB] flags=fmF perm=r
> offset=0x00012000 path=/zpool.so
> 0x0000100000082000 0x0000100000083000 [4.0 kB] flags=fmF perm=r
> offset=0x00018000 path=/zpool.so
> 0x0000100000083000 0x0000100000084000 [4.0 kB] flags=fmF perm=rw
> offset=0x00019000 path=/zpool.so
> 0x0000100000084000 0x0000100000086000 [8.0 kB] flags=f perm=rw
> 0x0000200000000000 0x0000200000001000 [4.0 kB] flags=p perm=none
> 0x0000200000001000 0x0000200000002000 [4.0 kB] flags=p perm=none
> 0x0000200000002000 0x0000200000101000 [1020.0 kB] flags=p perm=rw
> 0x0000200000101000 0x0000200000102000 [4.0 kB] flags=p perm=none
> 0x0000200000102000 0x0000200000201000 [1020.0 kB] flags=p perm=rw
> 0x0000800000000000 0x0000800000000000 [0.0 kB] flags=none perm=none
> (gdb) bt
> #0 abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: %d)\n")
> at runtime.cc:105
> #1 0x000000004023c75b in __assert_fail (expr=expr@entry=0x406742f8
> "ef->rflags & processor::rflags_if", file=file@entry=0x40674325
> "arch/x64/mmu.cc", line=line@entry=38, func=func@entry=0x4067431a
> "page_fault") at runtime.cc:139
> #2 0x0000000040399305 in page_fault (ef=0xffff800000015048) at
> arch/x64/arch-cpu.hh:107
> #3 <signal handler called>
> #4 0x000000004035ca19 in elf::object::symtab_len (this=0xffffa00000f19a00)
> at core/elf.cc:983
> #5 0x000000004035cad8 in elf::object::lookup_addr
> (this=0xffffa00000f19a00, addr=addr@entry=0x1000000254ce <is_mounted+30>)
> at core/elf.cc:1015
> #6 0x000000004035cca7 in elf::program::<lambda(const
> elf::program::modules_list&)>::operator() (__closure=<synthetic pointer>,
> __closure=<synthetic pointer>, ml=...) at core/elf.cc:1620
> #7 elf::program::with_modules<elf::program::lookup_addr(void
> const*)::<lambda(const elf::program::modules_list&)> > (f=...,
> this=0xffffa00000097e70) at include/osv/elf.hh:702
> #8 elf::program::lookup_addr (this=0xffffa00000097e70,
> addr=addr@entry=0x1000000254ce <is_mounted+30>) at core/elf.cc:1617
> #9 0x00000000404367cc in osv::lookup_name_demangled
> (addr=addr@entry=0x1000000254ce <is_mounted+30>,
> buf=buf@entry=0xffff8000012156d0 "???+19929103", len=len@entry=1024) at
> core/demangle.cc:47
> #10 0x000000004023c540 in print_backtrace () at runtime.cc:85
> #11 0x000000004023c714 in abort (fmt=fmt@entry=0x40645aff "Aborted\n") at
> runtime.cc:121
> #12 0x0000000040202989 in abort () at runtime.cc:98
> #13 0x0000000040345934 in mmu::vm_sigsegv (ef=0xffff800001216068,
> addr=<optimized out>) at core/mmu.cc:1314
> #14 mmu::vm_sigsegv (addr=<optimized out>, ef=0xffff800001216068) at
> core/mmu.cc:1308
> #15 0x000000004034782f in mmu::vm_fault (addr=addr@entry=17592186309800,
> ef=ef@entry=0xffff800001216068) at core/mmu.cc:1328
> #16 0x00000000403992a3 in page_fault (ef=0xffff800001216068) at
> arch/x64/mmu.cc:42
> #17 <signal handler called>
> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot
> (this=this@entry=0xffffa00000f19a00, sym=...,
> addr=addr@entry=0x100000040ca8 <[email protected]>, addend=addend@entry=0)
> at arch/x64/arch-elf.cc:172
> #19 0x0000000040361004 in elf::object::resolve_pltgot
> (this=0xffffa00000f19a00, index=<optimized out>) at core/elf.cc:843
> #20 0x0000000040361229 in elf_resolve_pltgot (index=308,
> obj=0xffffa00000f19a00) at core/elf.cc:1860
> #21 0x0000000040397d50 in __elf_resolve_pltgot () at arch/x64/elf-dl.S:47
> #22 0x00001000000254cf in is_mounted (zfs_hdl=0x134, special=<optimized
> out>, where=0x403f3377 <malloc(size_t)+71>) at
> bsd/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_mount.c:214
> #23 0xffff900000a99000 in ?? ()
> #24 0x0000000000000000 in ?? ()
>
> On Tuesday, December 8, 2020 at 8:53:52 AM UTC-7 [email protected] wrote:
>
>> It would be also nice to understand if we are crashing on the 1st
>> arch_relocate_jump_slot() for libfzs.so or is it a specific JUMP_SLOT that
>> causes this crash?
>>
>> On Tuesday, December 8, 2020 at 10:39:06 AM UTC-5 Waldek Kozaczuk wrote:
>>
>>> After you connect with gdb can you run 'osv mmap' and send us the
>>> output. Make sure you run 'osv syms' before it and dump backtrace after.
>>> Please see https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>>> for any details.
>>>
>>> BTW can you build and run OSv ZFS image on the host without NIX? As I
>>> understand NIX is really just a layer on top of any Linux distribution, no?
>>> I am afraid I do not still understand what exactly NiX is I guess.
>>>
>>>
>>> On Monday, December 7, 2020 at 2:58:40 PM UTC-5 Matthew Kenigsberg wrote:
>>>
>>>> (gdb) frame 18
>>>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot
>>>> (this=this@entry=0xffffa0000110fa00, sym=...,
>>>> addr=addr@entry=0x100000040ca8, addend=addend@entry=0) at
>>>> arch/x64/arch-elf.cc:172
>>>> 172 *static_cast<void**>(addr) = sym.relocated_addr();
>>>> (gdb) print _pathname
>>>> $14 = {static npos = 18446744073709551615,
>>>> _M_dataplus = {<std::allocator<char>> =
>>>> {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
>>>> _M_p = 0xffffa0000110fa30 "/libzfs.so"}, _M_string_length = 10, {
>>>> _M_local_buf = "/libzfs.so\000\000\000\000\000",
>>>> _M_allocated_capacity = 3347131623889529903}}
>>>>
>>>> Also been wondering if nix using nonstandard paths is causing problems,
>>>> like for libc:
>>>> [nix-shell:~/osv/build/release]$ ldd libzfs.so
>>>> linux-vdso.so.1 (0x00007ffcedbb9000)
>>>> libuutil.so => not found
>>>> libc.so.6 =>
>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libc.so.6
>>>> (0x00007f7594f38000)
>>>>
>>>>
>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib64/ld-linux-x86-64.so.2
>>>>
>>>> (0x00007f7595131000)
>>>> On Sunday, December 6, 2020 at 8:43:10 AM UTC-7 [email protected]
>>>> wrote:
>>>>
>>>>> It might be easier to simply print '_pathname' value if you switch to
>>>>> the right frame in gdb. It would be nice to confirm that the problem we
>>>>> have is with zpool.so and that might lead to understanding why this crash
>>>>> happens. Maybe the is something wrong with building zpool.so.
>>>>>
>>>>> BTW based on this fragment of the stacktrace:
>>>>>
>>>>> #6 0x000000004035cb07 in elf::program::<lambda(const
>>>>> elf::program::modules_list&)>::operator() (
>>>>> __closure=<synthetic pointer>, __closure=<synthetic pointer>,
>>>>> ml=...) at core/elf.cc:1620
>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void
>>>>> const*)::<lambda(const elf::program::modules_list&)> >
>>>>> (f=..., this=0xffffa00000097e70) at include/osv/elf.hh:702
>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70,
>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617
>>>>> #9 0x00000000404357cc in osv::lookup_name_demangled
>>>>> (addr=addr@entry=0x1000000254ce,
>>>>> buf=buf@entry=0xffff8000012146d0 "???+19630095",
>>>>> len=len@entry=1024) at core/demangle.cc:47
>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85
>>>>>
>>>>> It seems we have a bug (or need of improvement) in print_backtrace()
>>>>> to make it NOT try to demangle names like "???+19630095" which causes
>>>>> follow-up fault.
>>>>>
>>>>> At the same time, it is strange that we crash at line 983 which seems
>>>>> to indicate something goes wrong when processing zpool.so.
>>>>>
>>>>> 981 if (dynamic_exists(DT_HASH)) {
>>>>>
>>>>> 982 auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH);
>>>>>
>>>>> *983 return hashtab[1];*
>>>>>
>>>>> 984 }
>>>>>
>>>>> On Sunday, December 6, 2020 at 10:06:21 AM UTC-5 Waldek Kozaczuk wrote:
>>>>>
>>>>>> Can you run the ROFS image you built? Also as I understand it NIX is
>>>>>> a package manager but what Linux distribution are you using?
>>>>>>
>>>>>> As far as ZFS goes could you enable ELF debugging - change this line:
>>>>>>
>>>>>> conf-debug_elf=0
>>>>>>
>>>>>> To
>>>>>>
>>>>>> conf-debug_elf=1
>>>>>>
>>>>>> In conf/base.mk, delete core/elf.o and force rebuild the kernel. I
>>>>>> think you may also need to change the script upload_manifest.py to
>>>>>> peeped
>>>>>> ‘—verbose’ to the command line with cpiod.so
>>>>>>
>>>>>> It should show more info about elf loading. It may still be necessary
>>>>>> to add extra printouts to capture which exact elf it is crashing on in
>>>>>> arch_relocate_jump().
>>>>>>
>>>>>> In worst case I would need a copy of your loader-stripped.elf and
>>>>>> possibly all the other files like cpiod.so, zfs.so that go into the
>>>>>> bootfs
>>>>>> part of the image.
>>>>>>
>>>>>> Regards,
>>>>>> Waldek
>>>>>>
>>>>>>
>>>>>> On Sat, Dec 5, 2020 at 19:31 Matthew Kenigsberg <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> After forcing it to use the right path for libz.so.1, it's working
>>>>>>> with rofs, but still having the same issue when using zfs, even after I
>>>>>>> correct the path for libz.
>>>>>>>
>>>>>>> On Saturday, December 5, 2020 at 5:18:37 PM UTC-7 Matthew Kenigsberg
>>>>>>> wrote:
>>>>>>>
>>>>>>>> gcc version 9.3.0 (GCC)
>>>>>>>> QEMU emulator version 5.1.0
>>>>>>>>
>>>>>>>> Running with fs=rofs I get the error:
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 369, in
>>>>>>>> <module>
>>>>>>>> main()
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 366, in
>>>>>>>> main
>>>>>>>> gen_image(outfile, manifest)
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 269, in
>>>>>>>> gen_image
>>>>>>>> system_structure_block, bytes_written = write_fs(fp, manifest)
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 246, in
>>>>>>>> write_fs
>>>>>>>> count, directory_entries_index = write_dir(fp,
>>>>>>>> manifest.get(''), '', manifest)
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 207, in
>>>>>>>> write_dir
>>>>>>>> count, directory_entries_index = write_dir(fp, val, dirpath +
>>>>>>>> '/' + entry, manifest)
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 207, in
>>>>>>>> write_dir
>>>>>>>> count, directory_entries_index = write_dir(fp, val, dirpath +
>>>>>>>> '/' + entry, manifest)
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 222, in
>>>>>>>> write_dir
>>>>>>>> inode.count = write_file(fp, val)
>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 164, in
>>>>>>>> write_file
>>>>>>>> with open(path, 'rb') as f:
>>>>>>>> FileNotFoundError: [Errno 2] No such file or directory: 'libz.so.1'
>>>>>>>>
>>>>>>>> I think that's from this line in usr.manifest?
>>>>>>>> /usr/lib/libz.so.1: libz.so.1
>>>>>>>>
>>>>>>>> Don't have zlib in the manifest without fs=rofs, and I think zpool
>>>>>>>> uses it?
>>>>>>>>
>>>>>>>> Looking into it...
>>>>>>>> On Saturday, December 5, 2020 at 4:36:20 PM UTC-7
>>>>>>>> [email protected] wrote:
>>>>>>>>
>>>>>>>>> I can not reproduce it on Ubuntu 20.20 neither Fedora 33. Here is
>>>>>>>>> the code fragment where it happens:
>>>>>>>>>
>>>>>>>>> 169 bool object::arch_relocate_jump_slot(symbol_module& sym, void
>>>>>>>>> *addr, Elf64_Sxword addend)
>>>>>>>>>
>>>>>>>>> 170 {
>>>>>>>>>
>>>>>>>>> 171 if (sym.symbol) {
>>>>>>>>>
>>>>>>>>> 172 *static_cast<void**>(addr) = sym.relocated_addr();
>>>>>>>>>
>>>>>>>>> 173 return true;
>>>>>>>>>
>>>>>>>>> 174 } else {
>>>>>>>>>
>>>>>>>>> 175 return false;
>>>>>>>>>
>>>>>>>>> 176 }
>>>>>>>>>
>>>>>>>>> 177 }
>>>>>>>>> It looks like writing at the addr 0x100000040ca8 in line 172
>>>>>>>>> caused the fault. Why?
>>>>>>>>>
>>>>>>>>> And then the 2nd page fault in the gdb backtrace as the 1st one
>>>>>>>>> was being handled (not sure if that is a bug or just a state of
>>>>>>>>> loading of
>>>>>>>>> a program).
>>>>>>>>>
>>>>>>>>> 981 if (dynamic_exists(DT_HASH)) {
>>>>>>>>>
>>>>>>>>> 982 auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH);
>>>>>>>>>
>>>>>>>>> 983 return hashtab[1];
>>>>>>>>>
>>>>>>>>> 984 }
>>>>>>>>> Is something wrong with the elf files cpiod.so, mkfs.so or zfs.so
>>>>>>>>> or something?
>>>>>>>>>
>>>>>>>>> Can you try to do the same with ROFS?
>>>>>>>>>
>>>>>>>>> fs=rofs
>>>>>>>>> On Saturday, December 5, 2020 at 5:44:12 PM UTC-5 Matthew
>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>
>>>>>>>>>> Struggling to get scripts/build to run on NixOS because I'm
>>>>>>>>>> getting a page fault. NixOS does keep shared libraries in
>>>>>>>>>> nonstandard
>>>>>>>>>> locations, not sure if that's breaking something. More details
>>>>>>>>>> below, but
>>>>>>>>>> any ideas?
>>>>>>>>>>
>>>>>>>>>> As far as I can tell, the error is caused by
>>>>>>>>>> tools/mkfs/mkfs.cc:71:
>>>>>>>>>> run_cmd("/zpool.so", zpool_args);
>>>>>>>>>>
>>>>>>>>>> The error from scripts/build:
>>>>>>>>>>
>>>>>>>>>> OSv v0.55.0-145-g97f17a7a
>>>>>>>>>> eth0: 192.168.122.15
>>>>>>>>>> Booted up in 154.38 ms
>>>>>>>>>> Cmdline: /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/zfs/;
>>>>>>>>>> /zfs.so set compression=off osv
>>>>>>>>>> Running mkfs...
>>>>>>>>>> page fault outside application, addr: 0x0000100000040ca8
>>>>>>>>>> [registers]
>>>>>>>>>> RIP: 0x000000004039c25a
>>>>>>>>>> <elf::object::arch_relocate_jump_slot(elf::symbol_module&, void*,
>>>>>>>>>> long)+26>
>>>>>>>>>> RFL: 0x0000000000010202 CS: 0x0000000000000008 SS:
>>>>>>>>>> 0x0000000000000010
>>>>>>>>>> RAX: 0x000010000007a340 RBX: 0x0000100000040ca8 RCX:
>>>>>>>>>> 0x000010000006abb0 RDX: 0x0000000000000002
>>>>>>>>>> RSI: 0x00002000001f6f70 RDI: 0xffffa00001058c00 RBP:
>>>>>>>>>> 0x00002000001f6f30 R8: 0xffffa00000a68460
>>>>>>>>>> R9: 0xffffa00000f18da0 R10: 0x0000000000000000 R11:
>>>>>>>>>> 0x00000000409dd380 R12: 0xffffa00000f18c00
>>>>>>>>>> R13: 0xffffa00000f18da0 R14: 0x0000000000000000 R15:
>>>>>>>>>> 0x00000000409dd380 RSP: 0x00002000001f6f20
>>>>>>>>>> Aborted
>>>>>>>>>>
>>>>>>>>>> [backtrace]
>>>>>>>>>> 0x00000000403458d3 <???+1077172435>
>>>>>>>>>> 0x00000000403477ce <mmu::vm_fault(unsigned long,
>>>>>>>>>> exception_frame*)+350>
>>>>>>>>>> 0x0000000040398ba2 <page_fault+162>
>>>>>>>>>> 0x0000000040397a16 <???+1077508630>
>>>>>>>>>> 0x0000000040360a13 <elf::object::resolve_pltgot(unsigned int)+387>
>>>>>>>>>> 0x0000000040360c38 <elf_resolve_pltgot+56>
>>>>>>>>>> 0x000000004039764f <???+1077507663>
>>>>>>>>>> 0xffffa000012b880f <???+19630095>
>>>>>>>>>>
>>>>>>>>>> Trying to get a backtrace after connecting with gdb:
>>>>>>>>>> (gdb) bt
>>>>>>>>>> #0 abort (fmt=fmt@entry=0x40644b90 "Assertion failed: %s (%s:
>>>>>>>>>> %s: %d)\n") at runtime.cc:105
>>>>>>>>>> #1 0x000000004023c6fb in __assert_fail
>>>>>>>>>> (expr=expr@entry=0x40672cf8 "ef->rflags & processor::rflags_if",
>>>>>>>>>> file=file@entry=0x40672d25 "arch/x64/mmu.cc",
>>>>>>>>>> line=line@entry=38, func=func@entry=0x40672d1a "page_fault")
>>>>>>>>>> at runtime.cc:139
>>>>>>>>>> #2 0x0000000040398c05 in page_fault (ef=0xffff800000015048) at
>>>>>>>>>> arch/x64/arch-cpu.hh:107
>>>>>>>>>> #3 <signal handler called>
>>>>>>>>>> #4 0x000000004035c879 in elf::object::symtab_len
>>>>>>>>>> (this=0xffffa00000f18c00) at core/elf.cc:983
>>>>>>>>>> #5 0x000000004035c938 in elf::object::lookup_addr
>>>>>>>>>> (this=0xffffa00000f18c00, addr=addr@entry=0x1000000254ce)
>>>>>>>>>> at core/elf.cc:1015
>>>>>>>>>> #6 0x000000004035cb07 in elf::program::<lambda(const
>>>>>>>>>> elf::program::modules_list&)>::operator() (
>>>>>>>>>> __closure=<synthetic pointer>, __closure=<synthetic pointer>,
>>>>>>>>>> ml=...) at core/elf.cc:1620
>>>>>>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void
>>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> >
>>>>>>>>>> (f=..., this=0xffffa00000097e70) at include/osv/elf.hh:702
>>>>>>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70,
>>>>>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617
>>>>>>>>>> #9 0x00000000404357cc in osv::lookup_name_demangled
>>>>>>>>>> (addr=addr@entry=0x1000000254ce,
>>>>>>>>>> buf=buf@entry=0xffff8000012146d0 "???+19630095",
>>>>>>>>>> len=len@entry=1024) at core/demangle.cc:47
>>>>>>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85
>>>>>>>>>> #11 0x000000004023c6b4 in abort (fmt=fmt@entry=0x40644a9f
>>>>>>>>>> "Aborted\n") at runtime.cc:121
>>>>>>>>>> #12 0x0000000040202989 in abort () at runtime.cc:98
>>>>>>>>>> #13 0x00000000403458d4 in mmu::vm_sigsegv (ef=0xffff800001215068,
>>>>>>>>>> addr=<optimized out>) at core/mmu.cc:1314
>>>>>>>>>> #14 mmu::vm_sigsegv (addr=<optimized out>, ef=0xffff800001215068)
>>>>>>>>>> at core/mmu.cc:1308
>>>>>>>>>> #15 0x00000000403477cf in mmu::vm_fault
>>>>>>>>>> (addr=addr@entry=17592186309800, ef=ef@entry=0xffff800001215068)
>>>>>>>>>> at core/mmu.cc:1328
>>>>>>>>>> #16 0x0000000040398ba3 in page_fault (ef=0xffff800001215068) at
>>>>>>>>>> arch/x64/mmu.cc:42
>>>>>>>>>> #17 <signal handler called>
>>>>>>>>>> #18 0x000000004039c25a in elf::object::arch_relocate_jump_slot
>>>>>>>>>> (this=this@entry=0xffffa00000f18c00, sym=...,
>>>>>>>>>> addr=addr@entry=0x100000040ca8, addend=addend@entry=0) at
>>>>>>>>>> arch/x64/arch-elf.cc:172
>>>>>>>>>> #19 0x0000000040360a14 in elf::object::resolve_pltgot
>>>>>>>>>> (this=0xffffa00000f18c00, index=<optimized out>)
>>>>>>>>>> at core/elf.cc:843
>>>>>>>>>> #20 0x0000000040360c39 in elf_resolve_pltgot (index=308,
>>>>>>>>>> obj=0xffffa00000f18c00) at core/elf.cc:1860
>>>>>>>>>> #21 0x0000000040397650 in __elf_resolve_pltgot () at
>>>>>>>>>> arch/x64/elf-dl.S:47
>>>>>>>>>> #22 0x00001000000254cf in ?? ()
>>>>>>>>>> #23 0xffffa000012b8800 in ?? ()
>>>>>>>>>> #24 0x00002000001f74a0 in ?? ()
>>>>>>>>>> #25 0x00001000000254cf in ?? ()
>>>>>>>>>> #26 0x00002000001f7480 in ?? ()
>>>>>>>>>> #27 0x00000000403f241c in calloc (nmemb=<optimized out>,
>>>>>>>>>> size=<optimized out>) at core/mempool.cc:1811
>>>>>>>>>> #28 0xffff900000a98000 in ?? ()
>>>>>>>>>> #29 0x0000000000000000 in ?? ()
>>>>>>>>>> On Saturday, November 28, 2020 at 1:39:46 PM UTC-7 Matthew
>>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'll send something, might take a bit before I find time to work
>>>>>>>>>>> on it though.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Matthew
>>>>>>>>>>>
>>>>>>>>>>> On Saturday, November 28, 2020 at 1:11:11 PM UTC-7 Roman
>>>>>>>>>>> Shaposhnik wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Nov 24, 2020 at 8:03 AM Waldek Kozaczuk <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > Hey,
>>>>>>>>>>>> >
>>>>>>>>>>>> > Send a patch with a new app that could demonstrate it,
>>>>>>>>>>>> please, if you can. I would like to see it. Sounds like a nice
>>>>>>>>>>>> improvement.
>>>>>>>>>>>>
>>>>>>>>>>>> FWIW: I'd love to see it too -- been meaning to play with Nix
>>>>>>>>>>>> and this
>>>>>>>>>>>> gives me a perfect excuse ;-)
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Roman.
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>> the Google Groups "OSv Development" group.
>>>>>>> To unsubscribe from this topic, visit
>>>>>>> https://groups.google.com/d/topic/osv-dev/rhjHPr7OBEw/unsubscribe.
>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>> [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com
>>>>>>>
>>>>>>> <https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/1637b32e-ce86-46f6-b7f9-885d63d68601n%40googlegroups.com.
