My gdb is not the strongest but if I hbreak on arch_relocate_jump_slot
looks like _pathname is /libzfs.so, eventually /zpool.so, and then a single
/libzfs.so before continue hangs
On Tuesday, December 8, 2020 at 9:11:15 AM UTC-7 Matthew Kenigsberg wrote:
> Nix is a package manager, and NixOS is an operating system built
> completely around the package manager. So every library is stored somewhere
> in /nix/store, like for example on Nix there is never anything like
> /lib64/ld-linux-x86-64.so.
> It would be /nix/store/.../ld-linux-x86-64.so. I could install the package
> manager on a different OS, in which case I might have both /lib64 and
> /nix/store, but on NixOS I'll just have the latter. Does that make sense? Not
> sure if that's messing up something with linking. Guessing I can't
> reproduce the error on any other OS, but happy to try.
> On Tuesday, December 8, 2020 at 9:05:11 AM UTC-7 Matthew Kenigsberg wrote:
>
>> (gdb) connect
>> abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: %d)\n") at
>> runtime.cc:105
>> 105 do {} while (true);
>> (gdb) osv syms
>> manifest.find_file: path=/libvdso.so, found file=libvdso.so
>> /home/matthew/osv/build/release.x64/libvdso.so 0x100000000000
>> add symbol table from file
>> "/home/matthew/osv/build/release.x64/libvdso.so" at
>> .text_addr = 0x100000001040
>> .hash_addr = 0x1000000001c8
>> .gnu.hash_addr = 0x100000000200
>> .dynsym_addr = 0x100000000238
>> .dynstr_addr = 0x1000000002f8
>> .gnu.version_addr = 0x1000000003be
>> .gnu.version_d_addr = 0x1000000003d0
>> .rela.plt_addr = 0x100000000408
>> .plt_addr = 0x100000001000
>> .eh_frame_addr = 0x100000002000
>> .dynamic_addr = 0x100000003e60
>> .got_addr = 0x100000003fd0
>> .comment_addr = 0x100000000000
>> .debug_aranges_addr = 0x100000000000
>> .debug_info_addr = 0x100000000000
>> .debug_abbrev_addr = 0x100000000000
>> .debug_line_addr = 0x100000000000
>> .debug_str_addr = 0x100000000000
>> .debug_loc_addr = 0x100000000000
>> .symtab_addr = 0x100000000000
>> .strtab_addr = 0x100000000000
>> warning: section .comment not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .debug_aranges not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .debug_info not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .debug_abbrev not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .debug_line not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .debug_str not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .debug_loc not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .symtab not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> warning: section .strtab not found in
>> /home/matthew/osv/build/release.x64/libvdso.so
>> manifest.find_file: path=/tools/mkfs.so, found file=tools/mkfs/mkfs.so
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so 0x100000004000
>> add symbol table from file
>> "/home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so" at
>> .text_addr = 0x100000006250
>> .hash_addr = 0x100000004200
>> .gnu.hash_addr = 0x100000004360
>> .dynsym_addr = 0x1000000043c0
>> .dynstr_addr = 0x100000004840
>> .gnu.version_addr = 0x100000005092
>> .gnu.version_r_addr = 0x1000000050f8
>> .rela.dyn_addr = 0x100000005148
>> .rela.plt_addr = 0x100000005298
>> .init_addr = 0x100000006000
>> .plt_addr = 0x100000006020
>> .plt.got_addr = 0x100000006240
>> .fini_addr = 0x10000000737c
>> .rodata_addr = 0x100000008000
>> .eh_frame_hdr_addr = 0x10000000817c
>> .eh_frame_addr = 0x100000008210
>> .gcc_except_table_addr = 0x100000008530
>> .init_array_addr = 0x100000009c60
>> .fini_array_addr = 0x100000009c70
>> .dynamic_addr = 0x100000009c78
>> .got_addr = 0x100000009e98
>> .data_addr = 0x10000000a000
>> .bss_addr = 0x10000000a010
>> .comment_addr = 0x100000004000
>> .debug_aranges_addr = 0x100000004000
>> .debug_info_addr = 0x100000004000
>> .debug_abbrev_addr = 0x100000004000
>> --Type <RET> for more, q to quit, c to continue without paging--c
>> .debug_line_addr = 0x100000004000
>> .debug_str_addr = 0x100000004000
>> .debug_loc_addr = 0x100000004000
>> .debug_ranges_addr = 0x100000004000
>> .symtab_addr = 0x100000004000
>> .strtab_addr = 0x100000004000
>> warning: section .comment not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_aranges not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_info not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_abbrev not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_line not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_str not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_loc not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .debug_ranges not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .symtab not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> warning: section .strtab not found in
>> /home/matthew/osv/build/release.x64/tools/mkfs/mkfs.so
>> manifest.find_file: path=/libzfs.so, found file=libzfs.so
>> /home/matthew/osv/build/release.x64/libzfs.so 0x10000000b000
>> add symbol table from file
>> "/home/matthew/osv/build/release.x64/libzfs.so" at
>> .text_addr = 0x1000000178f0
>> .hash_addr = 0x10000000b200
>> .gnu.hash_addr = 0x10000000c318
>> .dynsym_addr = 0x10000000cd78
>> .dynstr_addr = 0x100000010300
>> .gnu.version_addr = 0x1000000124e2
>> .gnu.version_r_addr = 0x100000012958
>> .rela.dyn_addr = 0x1000000129b8
>> .rela.plt_addr = 0x1000000134b0
>> .init_addr = 0x100000016000
>> .plt_addr = 0x100000016020
>> .plt.got_addr = 0x1000000178e0
>> .fini_addr = 0x1000000340a0
>> .rodata_addr = 0x100000035000
>> .eh_frame_hdr_addr = 0x100000039768
>> .eh_frame_addr = 0x10000003a470
>> .init_array_addr = 0x100000040010
>> .fini_array_addr = 0x100000040018
>> .data.rel.ro_addr = 0x100000040020
>> .dynamic_addr = 0x1000000400e0
>> .got_addr = 0x1000000402f0
>> .data_addr = 0x100000041000
>> .bss_addr = 0x100000041820
>> .comment_addr = 0x10000000b000
>> .debug_aranges_addr = 0x10000000b000
>> .debug_info_addr = 0x10000000b000
>> .debug_abbrev_addr = 0x10000000b000
>> .debug_line_addr = 0x10000000b000
>> .debug_str_addr = 0x10000000b000
>> .debug_loc_addr = 0x10000000b000
>> .debug_ranges_addr = 0x10000000b000
>> .symtab_addr = 0x10000000b000
>> .strtab_addr = 0x10000000b000
>> warning: section .comment not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_aranges not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_info not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_abbrev not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_line not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_str not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_loc not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .debug_ranges not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .symtab not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> warning: section .strtab not found in
>> /home/matthew/osv/build/release.x64/libzfs.so
>> manifest.find_file: path=/libuutil.so, found file=libuutil.so
>> /home/matthew/osv/build/release.x64/libuutil.so 0x100000043000
>> add symbol table from file
>> "/home/matthew/osv/build/release.x64/libuutil.so" at
>> .text_addr = 0x1000000463c0
>> .hash_addr = 0x100000043200
>> .gnu.hash_addr = 0x100000043640
>> .dynsym_addr = 0x1000000438f8
>> .dynstr_addr = 0x100000044600
>> .gnu.version_addr = 0x100000044da4
>> .gnu.version_r_addr = 0x100000044ec0
>> .rela.dyn_addr = 0x100000044f00
>> .rela.plt_addr = 0x100000045068
>> .init_addr = 0x100000046000
>> .plt_addr = 0x100000046020
>> .plt.got_addr = 0x1000000463b0
>> .fini_addr = 0x100000049adc
>> .rodata_addr = 0x10000004a000
>> .eh_frame_hdr_addr = 0x10000004ac84
>> .eh_frame_addr = 0x10000004af90
>> .init_array_addr = 0x10000004dbd8
>> .fini_array_addr = 0x10000004dbe0
>> .dynamic_addr = 0x10000004dbe8
>> .got_addr = 0x10000004dde8
>> .data_addr = 0x10000004e000
>> .bss_addr = 0x10000004e260
>> .comment_addr = 0x100000043000
>> .debug_aranges_addr = 0x100000043000
>> .debug_info_addr = 0x100000043000
>> .debug_abbrev_addr = 0x100000043000
>> .debug_line_addr = 0x100000043000
>> .debug_str_addr = 0x100000043000
>> .debug_loc_addr = 0x100000043000
>> .debug_ranges_addr = 0x100000043000
>> .symtab_addr = 0x100000043000
>> .strtab_addr = 0x100000043000
>> warning: section .comment not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_aranges not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_info not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_abbrev not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_line not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_str not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_loc not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .debug_ranges not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .symtab not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> warning: section .strtab not found in
>> /home/matthew/osv/build/release.x64/libuutil.so
>> manifest.find_file: path=/usr/lib/libgcc_s.so.1, found
>> file=%(libgcc_s_dir)s/libgcc_s.so.1
>> ERROR: Unable to locate object file for: /usr/lib/libgcc_s.so.1
>> 0x10000004f000
>> manifest.find_file: path=/zpool.so, found file=zpool.so
>> /home/matthew/osv/build/release.x64/zpool.so 0x100000069000
>> add symbol table from file "/home/matthew/osv/build/release.x64/zpool.so"
>> at
>> .text_addr = 0x10000006ebc0
>> .hash_addr = 0x100000069200
>> .gnu.hash_addr = 0x100000069868
>> .dynsym_addr = 0x100000069968
>> .dynstr_addr = 0x10000006ad18
>> .gnu.version_addr = 0x10000006b958
>> .gnu.version_r_addr = 0x10000006bb00
>> .rela.dyn_addr = 0x10000006bb30
>> .rela.plt_addr = 0x10000006c208
>> .init_addr = 0x10000006e000
>> .plt_addr = 0x10000006e020
>> .plt.got_addr = 0x10000006ebb0
>> .fini_addr = 0x10000007a420
>> .rodata_addr = 0x10000007b000
>> .eh_frame_hdr_addr = 0x10000007fd30
>> .eh_frame_addr = 0x100000080028
>> .init_array_addr = 0x100000082758
>> .fini_array_addr = 0x100000082760
>> .dynamic_addr = 0x100000082768
>> .got_addr = 0x100000082978
>> .data_addr = 0x100000083000
>> .bss_addr = 0x100000083360
>> .comment_addr = 0x100000069000
>> .debug_aranges_addr = 0x100000069000
>> .debug_info_addr = 0x100000069000
>> .debug_abbrev_addr = 0x100000069000
>> .debug_line_addr = 0x100000069000
>> .debug_str_addr = 0x100000069000
>> .debug_loc_addr = 0x100000069000
>> .debug_ranges_addr = 0x100000069000
>> .symtab_addr = 0x100000069000
>> .strtab_addr = 0x100000069000
>> warning: section .comment not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_aranges not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_info not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_abbrev not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_line not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_str not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_loc not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .debug_ranges not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .symtab not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> warning: section .strtab not found in
>> /home/matthew/osv/build/release.x64/zpool.so
>> (gdb) osv mmap
>> 0x0000000000000000 0x0000000000000000 [0.0 kB] flags=none perm=none
>> 0x0000100000000000 0x0000100000001000 [4.0 kB] flags=fmF perm=r
>> offset=0x00000000 path=/libvdso.so
>> 0x0000100000001000 0x0000100000002000 [4.0 kB] flags=fmF perm=rx
>> offset=0x00001000 path=/libvdso.so
>> 0x0000100000002000 0x0000100000003000 [4.0 kB] flags=fmF perm=r
>> offset=0x00002000 path=/libvdso.so
>> 0x0000100000003000 0x0000100000004000 [4.0 kB] flags=fmF perm=r
>> offset=0x00002000 path=/libvdso.so
>> 0x0000100000004000 0x0000100000006000 [8.0 kB] flags=fmF perm=r
>> offset=0x00000000 path=/tools/mkfs.so
>> 0x0000100000006000 0x0000100000008000 [8.0 kB] flags=fmF perm=rx
>> offset=0x00002000 path=/tools/mkfs.so
>> 0x0000100000008000 0x0000100000009000 [4.0 kB] flags=fmF perm=r
>> offset=0x00004000 path=/tools/mkfs.so
>> 0x0000100000009000 0x000010000000a000 [4.0 kB] flags=fmF perm=r
>> offset=0x00004000 path=/tools/mkfs.so
>> 0x000010000000a000 0x000010000000b000 [4.0 kB] flags=fmF perm=rw
>> offset=0x00005000 path=/tools/mkfs.so
>> 0x000010000000b000 0x0000100000016000 [44.0 kB] flags=fmF perm=r
>> offset=0x00000000 path=/libzfs.so
>> 0x0000100000016000 0x0000100000035000 [124.0 kB] flags=fmF perm=rx
>> offset=0x0000b000 path=/libzfs.so
>> 0x0000100000035000 0x000010000003f000 [40.0 kB] flags=fmF perm=r
>> offset=0x0002a000 path=/libzfs.so
>> 0x0000100000040000 0x0000100000041000 [4.0 kB] flags=fmF perm=r
>> offset=0x00034000 path=/libzfs.so
>> 0x0000100000041000 0x0000100000042000 [4.0 kB] flags=fmF perm=rw
>> offset=0x00035000 path=/libzfs.so
>> 0x0000100000042000 0x0000100000043000 [4.0 kB] flags=f perm=rw
>> 0x0000100000043000 0x0000100000046000 [12.0 kB] flags=fmF perm=r
>> offset=0x00000000 path=/libuutil.so
>> 0x0000100000046000 0x000010000004a000 [16.0 kB] flags=fmF perm=rx
>> offset=0x00003000 path=/libuutil.so
>> 0x000010000004a000 0x000010000004c000 [8.0 kB] flags=fmF perm=r
>> offset=0x00007000 path=/libuutil.so
>> 0x000010000004d000 0x000010000004e000 [4.0 kB] flags=fmF perm=r
>> offset=0x00009000 path=/libuutil.so
>> 0x000010000004e000 0x000010000004f000 [4.0 kB] flags=fmF perm=rw
>> offset=0x0000a000 path=/libuutil.so
>> 0x000010000004f000 0x0000100000052000 [12.0 kB] flags=fmF perm=r
>> offset=0x00000000 path=/usr/lib/libgcc_s.so.1
>> 0x0000100000052000 0x0000100000063000 [68.0 kB] flags=fmF perm=rx
>> offset=0x00003000 path=/usr/lib/libgcc_s.so.1
>> 0x0000100000063000 0x0000100000067000 [16.0 kB] flags=fmF perm=r
>> offset=0x00014000 path=/usr/lib/libgcc_s.so.1
>> 0x0000100000067000 0x0000100000068000 [4.0 kB] flags=fmF perm=r
>> offset=0x00017000 path=/usr/lib/libgcc_s.so.1
>> 0x0000100000068000 0x0000100000069000 [4.0 kB] flags=fmF perm=rw
>> offset=0x00018000 path=/usr/lib/libgcc_s.so.1
>> 0x0000100000069000 0x000010000006e000 [20.0 kB] flags=fmF perm=r
>> offset=0x00000000 path=/zpool.so
>> 0x000010000006e000 0x000010000007b000 [52.0 kB] flags=fmF perm=rx
>> offset=0x00005000 path=/zpool.so
>> 0x000010000007b000 0x0000100000082000 [28.0 kB] flags=fmF perm=r
>> offset=0x00012000 path=/zpool.so
>> 0x0000100000082000 0x0000100000083000 [4.0 kB] flags=fmF perm=r
>> offset=0x00018000 path=/zpool.so
>> 0x0000100000083000 0x0000100000084000 [4.0 kB] flags=fmF perm=rw
>> offset=0x00019000 path=/zpool.so
>> 0x0000100000084000 0x0000100000086000 [8.0 kB] flags=f perm=rw
>> 0x0000200000000000 0x0000200000001000 [4.0 kB] flags=p perm=none
>> 0x0000200000001000 0x0000200000002000 [4.0 kB] flags=p perm=none
>> 0x0000200000002000 0x0000200000101000 [1020.0 kB] flags=p perm=rw
>> 0x0000200000101000 0x0000200000102000 [4.0 kB] flags=p perm=none
>> 0x0000200000102000 0x0000200000201000 [1020.0 kB] flags=p perm=rw
>> 0x0000800000000000 0x0000800000000000 [0.0 kB] flags=none perm=none
>> (gdb) bt
>> #0 abort (fmt=fmt@entry=0x40645bf0 "Assertion failed: %s (%s: %s: %d)\n")
>> at runtime.cc:105
>> #1 0x000000004023c75b in __assert_fail (expr=expr@entry=0x406742f8
>> "ef->rflags & processor::rflags_if", file=file@entry=0x40674325
>> "arch/x64/mmu.cc", line=line@entry=38, func=func@entry=0x4067431a
>> "page_fault") at runtime.cc:139
>> #2 0x0000000040399305 in page_fault (ef=0xffff800000015048) at
>> arch/x64/arch-cpu.hh:107
>> #3 <signal handler called>
>> #4 0x000000004035ca19 in elf::object::symtab_len
>> (this=0xffffa00000f19a00) at core/elf.cc:983
>> #5 0x000000004035cad8 in elf::object::lookup_addr
>> (this=0xffffa00000f19a00, addr=addr@entry=0x1000000254ce <is_mounted+30>)
>> at core/elf.cc:1015
>> #6 0x000000004035cca7 in elf::program::<lambda(const
>> elf::program::modules_list&)>::operator() (__closure=<synthetic pointer>,
>> __closure=<synthetic pointer>, ml=...) at core/elf.cc:1620
>> #7 elf::program::with_modules<elf::program::lookup_addr(void
>> const*)::<lambda(const elf::program::modules_list&)> > (f=...,
>> this=0xffffa00000097e70) at include/osv/elf.hh:702
>> #8 elf::program::lookup_addr (this=0xffffa00000097e70,
>> addr=addr@entry=0x1000000254ce <is_mounted+30>) at core/elf.cc:1617
>> #9 0x00000000404367cc in osv::lookup_name_demangled
>> (addr=addr@entry=0x1000000254ce <is_mounted+30>,
>> buf=buf@entry=0xffff8000012156d0 "???+19929103", len=len@entry=1024) at
>> core/demangle.cc:47
>> #10 0x000000004023c540 in print_backtrace () at runtime.cc:85
>> #11 0x000000004023c714 in abort (fmt=fmt@entry=0x40645aff "Aborted\n") at
>> runtime.cc:121
>> #12 0x0000000040202989 in abort () at runtime.cc:98
>> #13 0x0000000040345934 in mmu::vm_sigsegv (ef=0xffff800001216068,
>> addr=<optimized out>) at core/mmu.cc:1314
>> #14 mmu::vm_sigsegv (addr=<optimized out>, ef=0xffff800001216068) at
>> core/mmu.cc:1308
>> #15 0x000000004034782f in mmu::vm_fault (addr=addr@entry=17592186309800,
>> ef=ef@entry=0xffff800001216068) at core/mmu.cc:1328
>> #16 0x00000000403992a3 in page_fault (ef=0xffff800001216068) at
>> arch/x64/mmu.cc:42
>> #17 <signal handler called>
>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot
>> (this=this@entry=0xffffa00000f19a00, sym=...,
>> addr=addr@entry=0x100000040ca8 <[email protected]>, addend=addend@entry=0)
>> at arch/x64/arch-elf.cc:172
>> #19 0x0000000040361004 in elf::object::resolve_pltgot
>> (this=0xffffa00000f19a00, index=<optimized out>) at core/elf.cc:843
>> #20 0x0000000040361229 in elf_resolve_pltgot (index=308,
>> obj=0xffffa00000f19a00) at core/elf.cc:1860
>> #21 0x0000000040397d50 in __elf_resolve_pltgot () at arch/x64/elf-dl.S:47
>> #22 0x00001000000254cf in is_mounted (zfs_hdl=0x134, special=<optimized
>> out>, where=0x403f3377 <malloc(size_t)+71>) at
>> bsd/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_mount.c:214
>> #23 0xffff900000a99000 in ?? ()
>> #24 0x0000000000000000 in ?? ()
>>
>> On Tuesday, December 8, 2020 at 8:53:52 AM UTC-7 [email protected]
>> wrote:
>>
>>> It would be also nice to understand if we are crashing on the 1st
>>> arch_relocate_jump_slot() for libfzs.so or is it a specific JUMP_SLOT that
>>> causes this crash?
>>>
>>> On Tuesday, December 8, 2020 at 10:39:06 AM UTC-5 Waldek Kozaczuk wrote:
>>>
>>>> After you connect with gdb can you run 'osv mmap' and send us the
>>>> output. Make sure you run 'osv syms' before it and dump backtrace after.
>>>> Please see https://github.com/cloudius-systems/osv/wiki/Debugging-OSv
>>>> for any details.
>>>>
>>>> BTW can you build and run OSv ZFS image on the host without NIX? As I
>>>> understand NIX is really just a layer on top of any Linux distribution,
>>>> no?
>>>> I am afraid I do not still understand what exactly NiX is I guess.
>>>>
>>>>
>>>> On Monday, December 7, 2020 at 2:58:40 PM UTC-5 Matthew Kenigsberg
>>>> wrote:
>>>>
>>>>> (gdb) frame 18
>>>>> #18 0x000000004039c95a in elf::object::arch_relocate_jump_slot
>>>>> (this=this@entry=0xffffa0000110fa00, sym=...,
>>>>> addr=addr@entry=0x100000040ca8, addend=addend@entry=0) at
>>>>> arch/x64/arch-elf.cc:172
>>>>> 172 *static_cast<void**>(addr) = sym.relocated_addr();
>>>>> (gdb) print _pathname
>>>>> $14 = {static npos = 18446744073709551615,
>>>>> _M_dataplus = {<std::allocator<char>> =
>>>>> {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data
>>>>> fields>},
>>>>> _M_p = 0xffffa0000110fa30 "/libzfs.so"}, _M_string_length = 10, {
>>>>> _M_local_buf = "/libzfs.so\000\000\000\000\000",
>>>>> _M_allocated_capacity = 3347131623889529903}}
>>>>>
>>>>> Also been wondering if nix using nonstandard paths is causing
>>>>> problems, like for libc:
>>>>> [nix-shell:~/osv/build/release]$ ldd libzfs.so
>>>>> linux-vdso.so.1 (0x00007ffcedbb9000)
>>>>> libuutil.so => not found
>>>>> libc.so.6 =>
>>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libc.so.6
>>>>> (0x00007f7594f38000)
>>>>>
>>>>>
>>>>> /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib64/ld-linux-x86-64.so.2
>>>>>
>>>>> (0x00007f7595131000)
>>>>> On Sunday, December 6, 2020 at 8:43:10 AM UTC-7 [email protected]
>>>>> wrote:
>>>>>
>>>>>> It might be easier to simply print '_pathname' value if you switch to
>>>>>> the right frame in gdb. It would be nice to confirm that the problem we
>>>>>> have is with zpool.so and that might lead to understanding why this
>>>>>> crash
>>>>>> happens. Maybe the is something wrong with building zpool.so.
>>>>>>
>>>>>> BTW based on this fragment of the stacktrace:
>>>>>>
>>>>>> #6 0x000000004035cb07 in elf::program::<lambda(const
>>>>>> elf::program::modules_list&)>::operator() (
>>>>>> __closure=<synthetic pointer>, __closure=<synthetic pointer>,
>>>>>> ml=...) at core/elf.cc:1620
>>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void
>>>>>> const*)::<lambda(const elf::program::modules_list&)> >
>>>>>> (f=..., this=0xffffa00000097e70) at include/osv/elf.hh:702
>>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70,
>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617
>>>>>> #9 0x00000000404357cc in osv::lookup_name_demangled
>>>>>> (addr=addr@entry=0x1000000254ce,
>>>>>> buf=buf@entry=0xffff8000012146d0 "???+19630095",
>>>>>> len=len@entry=1024) at core/demangle.cc:47
>>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85
>>>>>>
>>>>>> It seems we have a bug (or need of improvement) in print_backtrace()
>>>>>> to make it NOT try to demangle names like "???+19630095" which causes
>>>>>> follow-up fault.
>>>>>>
>>>>>> At the same time, it is strange that we crash at line 983 which seems
>>>>>> to indicate something goes wrong when processing zpool.so.
>>>>>>
>>>>>> 981 if (dynamic_exists(DT_HASH)) {
>>>>>>
>>>>>> 982 auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH);
>>>>>>
>>>>>> *983 return hashtab[1];*
>>>>>>
>>>>>> 984 }
>>>>>>
>>>>>> On Sunday, December 6, 2020 at 10:06:21 AM UTC-5 Waldek Kozaczuk
>>>>>> wrote:
>>>>>>
>>>>>>> Can you run the ROFS image you built? Also as I understand it NIX is
>>>>>>> a package manager but what Linux distribution are you using?
>>>>>>>
>>>>>>> As far as ZFS goes could you enable ELF debugging - change this line:
>>>>>>>
>>>>>>> conf-debug_elf=0
>>>>>>>
>>>>>>> To
>>>>>>>
>>>>>>> conf-debug_elf=1
>>>>>>>
>>>>>>> In conf/base.mk, delete core/elf.o and force rebuild the kernel. I
>>>>>>> think you may also need to change the script upload_manifest.py to
>>>>>>> peeped
>>>>>>> ‘—verbose’ to the command line with cpiod.so
>>>>>>>
>>>>>>> It should show more info about elf loading. It may still be
>>>>>>> necessary to add extra printouts to capture which exact elf it is
>>>>>>> crashing
>>>>>>> on in arch_relocate_jump().
>>>>>>>
>>>>>>> In worst case I would need a copy of your loader-stripped.elf and
>>>>>>> possibly all the other files like cpiod.so, zfs.so that go into the
>>>>>>> bootfs
>>>>>>> part of the image.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Waldek
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Dec 5, 2020 at 19:31 Matthew Kenigsberg <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> After forcing it to use the right path for libz.so.1, it's working
>>>>>>>> with rofs, but still having the same issue when using zfs, even after
>>>>>>>> I
>>>>>>>> correct the path for libz.
>>>>>>>>
>>>>>>>> On Saturday, December 5, 2020 at 5:18:37 PM UTC-7 Matthew
>>>>>>>> Kenigsberg wrote:
>>>>>>>>
>>>>>>>>> gcc version 9.3.0 (GCC)
>>>>>>>>> QEMU emulator version 5.1.0
>>>>>>>>>
>>>>>>>>> Running with fs=rofs I get the error:
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 369, in
>>>>>>>>> <module>
>>>>>>>>> main()
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 366, in
>>>>>>>>> main
>>>>>>>>> gen_image(outfile, manifest)
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 269, in
>>>>>>>>> gen_image
>>>>>>>>> system_structure_block, bytes_written = write_fs(fp, manifest)
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 246, in
>>>>>>>>> write_fs
>>>>>>>>> count, directory_entries_index = write_dir(fp,
>>>>>>>>> manifest.get(''), '', manifest)
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 207, in
>>>>>>>>> write_dir
>>>>>>>>> count, directory_entries_index = write_dir(fp, val, dirpath +
>>>>>>>>> '/' + entry, manifest)
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 207, in
>>>>>>>>> write_dir
>>>>>>>>> count, directory_entries_index = write_dir(fp, val, dirpath +
>>>>>>>>> '/' + entry, manifest)
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 222, in
>>>>>>>>> write_dir
>>>>>>>>> inode.count = write_file(fp, val)
>>>>>>>>> File "/home/matthew/osv/scripts/gen-rofs-img.py", line 164, in
>>>>>>>>> write_file
>>>>>>>>> with open(path, 'rb') as f:
>>>>>>>>> FileNotFoundError: [Errno 2] No such file or directory: 'libz.so.1'
>>>>>>>>>
>>>>>>>>> I think that's from this line in usr.manifest?
>>>>>>>>> /usr/lib/libz.so.1: libz.so.1
>>>>>>>>>
>>>>>>>>> Don't have zlib in the manifest without fs=rofs, and I think zpool
>>>>>>>>> uses it?
>>>>>>>>>
>>>>>>>>> Looking into it...
>>>>>>>>> On Saturday, December 5, 2020 at 4:36:20 PM UTC-7
>>>>>>>>> [email protected] wrote:
>>>>>>>>>
>>>>>>>>>> I can not reproduce it on Ubuntu 20.20 neither Fedora 33. Here is
>>>>>>>>>> the code fragment where it happens:
>>>>>>>>>>
>>>>>>>>>> 169 bool object::arch_relocate_jump_slot(symbol_module& sym, void
>>>>>>>>>> *addr, Elf64_Sxword addend)
>>>>>>>>>>
>>>>>>>>>> 170 {
>>>>>>>>>>
>>>>>>>>>> 171 if (sym.symbol) {
>>>>>>>>>>
>>>>>>>>>> 172 *static_cast<void**>(addr) = sym.relocated_addr();
>>>>>>>>>>
>>>>>>>>>> 173 return true;
>>>>>>>>>>
>>>>>>>>>> 174 } else {
>>>>>>>>>>
>>>>>>>>>> 175 return false;
>>>>>>>>>>
>>>>>>>>>> 176 }
>>>>>>>>>>
>>>>>>>>>> 177 }
>>>>>>>>>> It looks like writing at the addr 0x100000040ca8 in line 172
>>>>>>>>>> caused the fault. Why?
>>>>>>>>>>
>>>>>>>>>> And then the 2nd page fault in the gdb backtrace as the 1st one
>>>>>>>>>> was being handled (not sure if that is a bug or just a state of
>>>>>>>>>> loading of
>>>>>>>>>> a program).
>>>>>>>>>>
>>>>>>>>>> 981 if (dynamic_exists(DT_HASH)) {
>>>>>>>>>>
>>>>>>>>>> 982 auto hashtab = dynamic_ptr<Elf64_Word>(DT_HASH);
>>>>>>>>>>
>>>>>>>>>> 983 return hashtab[1];
>>>>>>>>>>
>>>>>>>>>> 984 }
>>>>>>>>>> Is something wrong with the elf files cpiod.so, mkfs.so or zfs.so
>>>>>>>>>> or something?
>>>>>>>>>>
>>>>>>>>>> Can you try to do the same with ROFS?
>>>>>>>>>>
>>>>>>>>>> fs=rofs
>>>>>>>>>> On Saturday, December 5, 2020 at 5:44:12 PM UTC-5 Matthew
>>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>>
>>>>>>>>>>> Struggling to get scripts/build to run on NixOS because I'm
>>>>>>>>>>> getting a page fault. NixOS does keep shared libraries in
>>>>>>>>>>> nonstandard
>>>>>>>>>>> locations, not sure if that's breaking something. More details
>>>>>>>>>>> below, but
>>>>>>>>>>> any ideas?
>>>>>>>>>>>
>>>>>>>>>>> As far as I can tell, the error is caused by
>>>>>>>>>>> tools/mkfs/mkfs.cc:71:
>>>>>>>>>>> run_cmd("/zpool.so", zpool_args);
>>>>>>>>>>>
>>>>>>>>>>> The error from scripts/build:
>>>>>>>>>>>
>>>>>>>>>>> OSv v0.55.0-145-g97f17a7a
>>>>>>>>>>> eth0: 192.168.122.15
>>>>>>>>>>> Booted up in 154.38 ms
>>>>>>>>>>> Cmdline: /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/zfs/;
>>>>>>>>>>> /zfs.so set compression=off osv
>>>>>>>>>>> Running mkfs...
>>>>>>>>>>> page fault outside application, addr: 0x0000100000040ca8
>>>>>>>>>>> [registers]
>>>>>>>>>>> RIP: 0x000000004039c25a
>>>>>>>>>>> <elf::object::arch_relocate_jump_slot(elf::symbol_module&, void*,
>>>>>>>>>>> long)+26>
>>>>>>>>>>> RFL: 0x0000000000010202 CS: 0x0000000000000008 SS:
>>>>>>>>>>> 0x0000000000000010
>>>>>>>>>>> RAX: 0x000010000007a340 RBX: 0x0000100000040ca8 RCX:
>>>>>>>>>>> 0x000010000006abb0 RDX: 0x0000000000000002
>>>>>>>>>>> RSI: 0x00002000001f6f70 RDI: 0xffffa00001058c00 RBP:
>>>>>>>>>>> 0x00002000001f6f30 R8: 0xffffa00000a68460
>>>>>>>>>>> R9: 0xffffa00000f18da0 R10: 0x0000000000000000 R11:
>>>>>>>>>>> 0x00000000409dd380 R12: 0xffffa00000f18c00
>>>>>>>>>>> R13: 0xffffa00000f18da0 R14: 0x0000000000000000 R15:
>>>>>>>>>>> 0x00000000409dd380 RSP: 0x00002000001f6f20
>>>>>>>>>>> Aborted
>>>>>>>>>>>
>>>>>>>>>>> [backtrace]
>>>>>>>>>>> 0x00000000403458d3 <???+1077172435>
>>>>>>>>>>> 0x00000000403477ce <mmu::vm_fault(unsigned long,
>>>>>>>>>>> exception_frame*)+350>
>>>>>>>>>>> 0x0000000040398ba2 <page_fault+162>
>>>>>>>>>>> 0x0000000040397a16 <???+1077508630>
>>>>>>>>>>> 0x0000000040360a13 <elf::object::resolve_pltgot(unsigned
>>>>>>>>>>> int)+387>
>>>>>>>>>>> 0x0000000040360c38 <elf_resolve_pltgot+56>
>>>>>>>>>>> 0x000000004039764f <???+1077507663>
>>>>>>>>>>> 0xffffa000012b880f <???+19630095>
>>>>>>>>>>>
>>>>>>>>>>> Trying to get a backtrace after connecting with gdb:
>>>>>>>>>>> (gdb) bt
>>>>>>>>>>> #0 abort (fmt=fmt@entry=0x40644b90 "Assertion failed: %s (%s:
>>>>>>>>>>> %s: %d)\n") at runtime.cc:105
>>>>>>>>>>> #1 0x000000004023c6fb in __assert_fail
>>>>>>>>>>> (expr=expr@entry=0x40672cf8 "ef->rflags & processor::rflags_if",
>>>>>>>>>>> file=file@entry=0x40672d25 "arch/x64/mmu.cc",
>>>>>>>>>>> line=line@entry=38, func=func@entry=0x40672d1a "page_fault")
>>>>>>>>>>> at runtime.cc:139
>>>>>>>>>>> #2 0x0000000040398c05 in page_fault (ef=0xffff800000015048) at
>>>>>>>>>>> arch/x64/arch-cpu.hh:107
>>>>>>>>>>> #3 <signal handler called>
>>>>>>>>>>> #4 0x000000004035c879 in elf::object::symtab_len
>>>>>>>>>>> (this=0xffffa00000f18c00) at core/elf.cc:983
>>>>>>>>>>> #5 0x000000004035c938 in elf::object::lookup_addr
>>>>>>>>>>> (this=0xffffa00000f18c00, addr=addr@entry=0x1000000254ce)
>>>>>>>>>>> at core/elf.cc:1015
>>>>>>>>>>> #6 0x000000004035cb07 in elf::program::<lambda(const
>>>>>>>>>>> elf::program::modules_list&)>::operator() (
>>>>>>>>>>> __closure=<synthetic pointer>, __closure=<synthetic
>>>>>>>>>>> pointer>, ml=...) at core/elf.cc:1620
>>>>>>>>>>> #7 elf::program::with_modules<elf::program::lookup_addr(void
>>>>>>>>>>> const*)::<lambda(const elf::program::modules_list&)> >
>>>>>>>>>>> (f=..., this=0xffffa00000097e70) at include/osv/elf.hh:702
>>>>>>>>>>> #8 elf::program::lookup_addr (this=0xffffa00000097e70,
>>>>>>>>>>> addr=addr@entry=0x1000000254ce) at core/elf.cc:1617
>>>>>>>>>>> #9 0x00000000404357cc in osv::lookup_name_demangled
>>>>>>>>>>> (addr=addr@entry=0x1000000254ce,
>>>>>>>>>>> buf=buf@entry=0xffff8000012146d0 "???+19630095",
>>>>>>>>>>> len=len@entry=1024) at core/demangle.cc:47
>>>>>>>>>>> #10 0x000000004023c4e0 in print_backtrace () at runtime.cc:85
>>>>>>>>>>> #11 0x000000004023c6b4 in abort (fmt=fmt@entry=0x40644a9f
>>>>>>>>>>> "Aborted\n") at runtime.cc:121
>>>>>>>>>>> #12 0x0000000040202989 in abort () at runtime.cc:98
>>>>>>>>>>> #13 0x00000000403458d4 in mmu::vm_sigsegv
>>>>>>>>>>> (ef=0xffff800001215068, addr=<optimized out>) at core/mmu.cc:1314
>>>>>>>>>>> #14 mmu::vm_sigsegv (addr=<optimized out>,
>>>>>>>>>>> ef=0xffff800001215068) at core/mmu.cc:1308
>>>>>>>>>>> #15 0x00000000403477cf in mmu::vm_fault
>>>>>>>>>>> (addr=addr@entry=17592186309800, ef=ef@entry=0xffff800001215068)
>>>>>>>>>>> at core/mmu.cc:1328
>>>>>>>>>>> #16 0x0000000040398ba3 in page_fault (ef=0xffff800001215068) at
>>>>>>>>>>> arch/x64/mmu.cc:42
>>>>>>>>>>> #17 <signal handler called>
>>>>>>>>>>> #18 0x000000004039c25a in elf::object::arch_relocate_jump_slot
>>>>>>>>>>> (this=this@entry=0xffffa00000f18c00, sym=...,
>>>>>>>>>>> addr=addr@entry=0x100000040ca8, addend=addend@entry=0) at
>>>>>>>>>>> arch/x64/arch-elf.cc:172
>>>>>>>>>>> #19 0x0000000040360a14 in elf::object::resolve_pltgot
>>>>>>>>>>> (this=0xffffa00000f18c00, index=<optimized out>)
>>>>>>>>>>> at core/elf.cc:843
>>>>>>>>>>> #20 0x0000000040360c39 in elf_resolve_pltgot (index=308,
>>>>>>>>>>> obj=0xffffa00000f18c00) at core/elf.cc:1860
>>>>>>>>>>> #21 0x0000000040397650 in __elf_resolve_pltgot () at
>>>>>>>>>>> arch/x64/elf-dl.S:47
>>>>>>>>>>> #22 0x00001000000254cf in ?? ()
>>>>>>>>>>> #23 0xffffa000012b8800 in ?? ()
>>>>>>>>>>> #24 0x00002000001f74a0 in ?? ()
>>>>>>>>>>> #25 0x00001000000254cf in ?? ()
>>>>>>>>>>> #26 0x00002000001f7480 in ?? ()
>>>>>>>>>>> #27 0x00000000403f241c in calloc (nmemb=<optimized out>,
>>>>>>>>>>> size=<optimized out>) at core/mempool.cc:1811
>>>>>>>>>>> #28 0xffff900000a98000 in ?? ()
>>>>>>>>>>> #29 0x0000000000000000 in ?? ()
>>>>>>>>>>> On Saturday, November 28, 2020 at 1:39:46 PM UTC-7 Matthew
>>>>>>>>>>> Kenigsberg wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'll send something, might take a bit before I find time to
>>>>>>>>>>>> work on it though.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Matthew
>>>>>>>>>>>>
>>>>>>>>>>>> On Saturday, November 28, 2020 at 1:11:11 PM UTC-7 Roman
>>>>>>>>>>>> Shaposhnik wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Nov 24, 2020 at 8:03 AM Waldek Kozaczuk <
>>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Hey,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Send a patch with a new app that could demonstrate it,
>>>>>>>>>>>>> please, if you can. I would like to see it. Sounds like a nice
>>>>>>>>>>>>> improvement.
>>>>>>>>>>>>>
>>>>>>>>>>>>> FWIW: I'd love to see it too -- been meaning to play with Nix
>>>>>>>>>>>>> and this
>>>>>>>>>>>>> gives me a perfect excuse ;-)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Roman.
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to a topic in
>>>>>>>> the Google Groups "OSv Development" group.
>>>>>>>> To unsubscribe from this topic, visit
>>>>>>>> https://groups.google.com/d/topic/osv-dev/rhjHPr7OBEw/unsubscribe.
>>>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>>>> [email protected].
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/7913b79b-6c06-4f2a-95d3-9dc44e45eb45n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/116b6f68-bdc4-4998-968e-b8ad9a867744n%40googlegroups.com.
