Recently I have been playing with GraalVM (https://github.com/oracle/graal) to see if it is possible to run it on OSv. To that extent I created new OSv app - https://github.com/cloudius-systems/osv-apps/tree/master/graalvm-example. As you can see it has a simple bootstrap main.so that loads a shared library libhello.so generated by GraalVM (similar to golang). The checked in line to build main.so is actually incorrect and should be $(CC) -pie -o $@ $(CFLAGS) -I. main.c -L. -lhello -ldl
In any case the app crashes like so: page fault outside application, addr: 0x0000100000d3f000 [registers] RIP: 0x00000000002fcb0c <elf::object::arch_relocate_rela(unsigned int, unsigned int, void*, long)+348> RFL: 0x0000000000000206 CS: 0x0000000000000008 SS: 0x0000000000000010 RAX: 0x0000000000000001 RBX: 0xffffa00001841800 RCX: 0x0000100000d3f030 RDX: 0x0000000000000000 RSI: 0x0000000000000000 RDI: 0xffffa00001841800 RBP: 0x00002000000ff090 R8: 0x0000100000ea9bb8 R9: 0x0000000000000008 R10: 0x0000000000000050 R11: 0x0000000000000000 R12: 0x00000000006a9bb8 R13: 0x0000000000000008 R14: 0x0000000000000050 R15: 0x0000000000000000 RSP: 0x00002000000ff050 Aborted [backtrace] 0x0000000000298ea8 <???+2723496> 0x0000000000299f06 <mmu::vm_fault(unsigned long, exception_frame*)+294> 0x00000000002f9238 <page_fault+136> 0x00000000002f8086 <???+3113094> 0x00000000002ab5d7 <elf::object::relocate_rela()+263> 0x00000000002ae3b7 <elf::object::relocate()+199> 0x00000000002b1e32 <elf::program::load_object(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<std::shared_ptr<elf::object>, std::allocator<std::shared_ptr<elf::object> > >&)+1602> 0x00000000002b1088 <elf::object::load_needed(std::vector<std::shared_ptr<elf::object>, std::allocator<std::shared_ptr<elf::object> > >&)+520> 0x00000000002b1e26 <elf::program::load_object(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<std::shared_ptr<elf::object>, std::allocator<std::shared_ptr<elf::object> > >&)+1590> 0x00000000002b267a <elf::program::get_library(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+330> 0x0000000000385de1 <osv::application::application(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<0x0000000000386527 <osv::application::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>0x000000000038678a <osv::application::run(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)+90> 0x00000000002135f9 <do_main_thread(void*)+2601> 0x00000000003b7ce5 <???+3898597> After adding some debug statements I discovered that the page fault is caused by linker trying to write to area of memory-mapped file segment when relocating symbols (arch_relocate_rela) because the page is read-only per ELF permissions. When I simply disable enforcing the ELF write permission to make any segment always writable like so: @@ -328,18 +328,18 @@ void file::load_segment(const Elf64_Phdr& phdr) unsigned perm = 0; if (phdr.p_flags & PF_X) perm |= mmu::perm_exec; - if (phdr.p_flags & PF_W) + //if (phdr.p_flags & PF_W) perm |= mmu::perm_write; if (phdr.p_flags & PF_R) perm |= mmu::perm_read; the program executes just fine. What is also interesting when I build main as non-pie to run it on Linux it works fine as well: $(CC) -o $@ $(CFLAGS) -I. main.c -L. -lhello -ldl So it seems like Linux ELF loader is possibly more lenient than OSv or OSv has some bug. And possibly something with GraalVM generated ELF libhello.so is abnormal and the relocations should not be happening in segments marked as read-only. Or maybe OSv linker enforces the ELF permission (especially write ones) to early and it needs to be delayed after the relocation process is finished by doing mprotect(). BTW I have added some debug statements that print interesting stuff when OSv tries to link: [/main.so]:Loading segment with base:0x100000400000 at:0x100000400000 of size:4096 from file at offset:0 which is locked:0 [/main.so]:Mmapped file at: 0x100000400000 with permissions - exec:1, write:0, read:4 [/main.so]:Loading segment with base:0x100000400000 at:0x100000600000 of size:8192 from file at offset:0 which is locked:0 [/main.so]:Mmapped file at: 0x100000600000 with permissions - exec:0, write:2, read:4 [/libhello.so]:Loading segment with base:0x100000800000 at:0x100000800000 of size:7806976 from file at offset:0 which is locked:0 [/libhello.so]:Mmapped file at: 0x100000800000 with permissions - exec:1, write:0, read:4 [/libhello.so]:Loading segment with base:0x100000800000 at:0x100001171000 of size:884736 from file at offset:7802880 which is locked:0 [/libhello.so]:Mmapped file at: 0x100001171000 with permissions - exec:0, write:2, read:4 [/libhello.so]:object::relocate_rela object::relocate_rela: base:0x100000800000, sym:0, addr:0x100000d3f030, offset:53f030 --> Was vma found:1, access fault:1 page fault outside application, addr: 0x0000100000d3f000 [registers] RIP: 0x00000000002fcb0c <elf::object::arch_relocate_rela(unsigned int, unsigned int, void*, long)+348> RFL: 0x0000000000000206 CS: 0x0000000000000008 SS: 0x0000000000000010 RAX: 0x0000000000000001 RBX: 0xffffa00001841800 RCX: 0x0000100000d3f030 RDX: 0x0000000000000000 RSI: 0x0000000000000000 RDI: 0xffffa00001841800 RBP: 0x00002000000ff090 R8: 0x0000100000ea9bb8 R9: 0x0000000000000008 R10: 0x0000000000000050 R11: 0x0000000000000000 R12: 0x00000000006a9bb8 R13: 0x0000000000000008 R14: 0x0000000000000050 R15: 0x0000000000000000 RSP: 0x00002000000ff050 Aborted And some readelf info: readelf -e libhello.so .... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x00000000007717dc 0x00000000007717dc R E 200000 LOAD 0x0000000000771dd0 0x0000000000971dd0 0x0000000000971dd0 0x00000000000d6730 0x00000000000d6738 RW 200000 DYNAMIC 0x0000000000771de8 0x0000000000971de8 0x0000000000971de8 0x00000000000001f0 0x00000000000001f0 RW 8 NOTE 0x00000000000001c8 0x00000000000001c8 0x00000000000001c8 0x0000000000000024 0x0000000000000024 R 4 GNU_EH_FRAME 0x0000000000771740 0x0000000000771740 0x0000000000771740 0x0000000000000024 0x0000000000000024 R 4 GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 RWE 10 GNU_RELRO 0x0000000000771dd0 0x0000000000971dd0 0x0000000000971dd0 0x0000000000000230 0x0000000000000230 R 1 The 0x100001171000 address falls within 1st non-writable segment (I think). Lastly I was comparing it to golang example that obviously works and similar info is like so: [/hello.so]:Loading segment with base:0x100000c00000 at:0x100000c00000 of size:1032192 from file at offset:0 which is locked:0 [/hello.so]:Mmapped file at: 0x100000c00000 with permissions - exec:1, write:0, read:4 [/hello.so]:Loading segment with base:0x100000c00000 at:0x100000efc000 of size:675840 from file at offset:1032192 which is locked:0 [/hello.so]:Mmapped file at: 0x100000efc000 with permissions - exec:0, write:2, read:4 [/hello.so]:object::relocate_rela object::relocate_rela: base:0x100000c00000, sym:0, addr:0x100000efc300, offset:2fc300 object::relocate_rela: base:0x100000c00000, sym:0, addr:0x100000efc308, offset:2fc308 object::relocate_rela: base:0x100000c00000, sym:0, addr:0x100000efc310, offset:2fc310 object::relocate_rela: base:0x100000c00000, sym:0, addr:0x100000f0eeb0, offset:30eeb0 .... Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x00000000000fbc58 0x00000000000fbc58 R E 200000 LOAD 0x00000000000fc300 0x00000000002fc300 0x00000000002fc300 0x00000000000a4184 0x00000000000c4ef8 RW 200000 DYNAMIC 0x000000000018ddb8 0x000000000038ddb8 0x000000000038ddb8 0x0000000000000200 0x0000000000000200 RW 8 NOTE 0x0000000000000238 0x0000000000000238 0x0000000000000238 0x0000000000000024 0x0000000000000024 R 4 NOTE 0x00000000000fbc20 0x00000000000fbc20 0x00000000000fbc20 0x0000000000000038 0x0000000000000038 R 20 TLS 0x00000000000fc300 0x00000000002fc300 0x00000000002fc300 0x0000000000000000 0x0000000000000008 R 8 GNU_EH_FRAME 0x00000000000fb818 0x00000000000fb818 0x00000000000fb818 0x00000000000000ac 0x00000000000000ac R 4 GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 0x0000000000000000 RW 10 GNU_RELRO 0x00000000000fc300 0x00000000002fc300 0x00000000002fc300 0x0000000000091d00 0x0000000000091d00 R 1 Any ideas as to what is wrong are greatly appreciated. Regards, Waldek -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.