Hmm, this might be a protobuf problem. I switch to another version of protobuf that matches what tensorflow uses. That seems to make the problem go away.
On Fri, Jan 24, 2020 at 12:10 AM zhiting zhu <zhiti...@cs.utexas.edu> wrote: > I feel it might be not the specific library's problem but osv's problem. > If I add > from google.protobuf.pyext import _message > before import TensorFlow, that library could be loaded without any > problem. But it then fails at the other places which could be successfully > loaded before the addition: > > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing DT_INIT function > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Finished executing DT_INIT function > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 565 DT_INIT_ARRAYSZ functions > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 0 init func 0x1000246e00f0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 1 init func 0x1000246e04b0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 2 init func 0x1000246d31e0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 3 init func 0x1000246d3740 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 4 init func 0x1000246d3770 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 5 init func 0x1000246d37a0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 6 init func 0x1000246d37d0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 7 init func 0x1000246d3800 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 8 init func 0x1000246d3830 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 9 init func 0x1000246d3860 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 10 init func 0x1000246d3890 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 11 init func 0x1000246d38c0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 12 init func 0x1000246d38f0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 13 init func 0x1000246d3920 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 14 init func 0x1000246d3950 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 15 init func 0x1000246d3980 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 16 init func 0x1000246d39b0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 17 init func 0x1000246d39e0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 18 init func 0x1000246d3a10 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 19 init func 0x1000246d3a40 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 20 init func 0x1000246d3a70 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 21 init func 0x1000246d3aa0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 22 init func 0x1000246d3ad0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 23 init func 0x1000246d3b00 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 24 init func 0x1000246d3b30 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 25 init func 0x1000246d3b60 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 26 init func 0x1000246d3b90 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 27 init func 0x1000246d3bc0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 28 init func 0x1000246d3bf0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 29 init func 0x1000246d3c20 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 30 init func 0x1000246d3c50 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 31 init func 0x1000246d3c80 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 32 init func 0x1000246d3cc0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 33 init func 0x1000246d3cf0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 34 init func 0x1000246d3d20 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 35 init func 0x1000246d3d50 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 36 init func 0x1000246d3d80 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 37 init func 0x1000246d3db0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 38 init func 0x1000246d3de0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 39 init func 0x1000246d3e10 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 40 init func 0x1000246d3e40 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 41 init func 0x1000246d3e70 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 42 init func 0x1000246d3ea0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 43 init func 0x1000246d3ed0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 44 init func 0x1000246d3f00 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 45 init func 0x1000246d3f30 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 46 init func 0x1000246d3f60 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 47 init func 0x1000246d3f90 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 48 init func 0x1000246d3fc0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 49 init func 0x1000246d3ff0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 50 init func 0x1000246d4020 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 51 init func 0x1000246d4050 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 52 init func 0x1000246d4080 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 53 init func 0x1000246d40b0 > ELF [tid:222, /lib/python3.6/tensorflow/libtensorflow_framework.so.1]: > Executing 54 init func 0x1000246d42a0 > Aborted > > [backtrace] > 0x0000000040463aeb <osv::generate_signal(siginfo&, exception_frame*)+59> > 0x0000000040463b5a <osv::handle_mmap_fault(unsigned long, int, > exception_frame*)+26> > 0x000000004032f3e9 <mmu::vm_fault(unsigned long, exception_frame*)+185> > 0x000000004038f786 <page_fault+166> > 0x000000004038e5c6 <???+1077470662> > > gdb outputs: > > #0 0x000000004039100a in processor::cli_hlt () at > arch/x64/processor.hh:247 > #1 nmi (ef=0xffff8001000c8068) at arch/x64/exceptions.cc:306 > #2 <signal handler called> > #3 sched::cpu::do_idle (this=this@entry=0xffff80000001a040) at > core/sched.cc:405 > #4 0x00000000403efd48 in sched::cpu::idle (this=0xffff80000001a040) at > core/sched.cc:423 > #5 0x00000000403efd6c in sched::cpu::<lambda()>::operator() > (__closure=<optimized out>) at core/sched.cc:165 > #6 std::_Function_handler<void(), > sched::cpu::init_idle_thread()::<lambda()> >::_M_invoke(const > std::_Any_data &) (__functor=...) > at /usr/include/c++/7/bits/std_function.h:316 > #7 0x00000000403efeb7 in sched::thread_main_c (t=0xffff8001000c3040) at > arch/x64/arch-switch.hh:321 > #8 0x000000004038f543 in thread_main () at arch/x64/entry.S:113 > > which doesn't correspond to a specific place in the library. > > The full log is attached. > > > > On Thu, Jan 23, 2020 at 6:01 PM zhiting zhu <zhiti...@cs.utexas.edu> > wrote: > >> I get the dump from gdb: >> >> #0 0x00000000403965f2 in processor::cli_hlt () at >> arch/x64/processor.hh:247 >> #1 arch::halt_no_interrupts () at arch/x64/arch.hh:48 >> #2 osv::halt () at arch/x64/power.cc:26 >> #3 0x0000000040221b14 in abort (fmt=fmt@entry=0x405ff34b "Aborted\n") >> at runtime.cc:132 >> #4 0x0000000040221b32 in abort () at runtime.cc:98 >> #5 0x0000000040463aec in osv::generate_signal (siginfo=..., >> ef=0xffff800102932068) at libc/signal.cc:124 >> #6 0x0000000040463b5b in osv::handle_mmap_fault (addr=<optimized out>, >> sig=<optimized out>, ef=<optimized out>) at libc/signal.cc:139 >> #7 0x000000004032f3ea in mmu::vm_fault (addr=<optimized out>, addr@entry=1, >> ef=ef@entry=0xffff800102932068) at core/mmu.cc:1336 >> #8 0x000000004038f787 in page_fault (ef=0xffff800102932068) at >> arch/x64/mmu.cc:42 >> #9 <signal handler called> >> #10 0x000010003896052a in >> google::protobuf::internal::AddDescriptors(google::protobuf::internal::DescriptorTable >> const*) () >> #11 0x00001000388aceb6 in ?? () >> #12 0x000000004033ff9a in elf::object::run_init_funcs >> (this=0xffffa0011ad8c800, argc=argc@entry=0, argv=argv@entry=0x0) at >> core/elf.cc:1175 >> #13 0x000000004034102b in elf::program::init_library >> (this=this@entry=0xffffa000001fbdf0, >> argc=argc@entry=0, argv=argv@entry=0x0) at core/elf.cc:1486 >> #14 0x00000000403473ac in elf::program::get_library >> (this=this@entry=0xffffa000001fbdf0, >> name="/lib/python3.6/google/protobuf/pyext/_ >> message.cpython-36m-x86_64-linux-gnu.so", >> extra_path=std::vector of length 0, capacity 0, >> delay_init=delay_init@entry=false) at core/elf.cc:1465 >> #15 0x0000000040462f4a in dlopen (filename=0x200006f47520 >> "/lib/python3.6/google/protobuf/pyext/_ >> message.cpython-36m-x86_64-linux-gnu.so", flags=<optimized out>) at >> libc/dlfcn.cc:54 >> #16 0x0000100000937229 in _PyImport_FindSharedFuncptr () >> #17 0x000010000095dba7 in _PyImport_LoadDynamicModuleWithSpec () >> >> It seems the problem is this function: >> google::protobuf::internal::AddDescriptors. Unfortunately, tensorflow >> library strips the debug info even through I add -g when I compile it. >> >> >> >> On Thu, Jan 23, 2020 at 5:33 PM Waldek Kozaczuk <jwkozac...@gmail.com> >> wrote: >> >>> Yeah that is pretty big image given that the default memory size if 4GB. >>> I wonder if increasing the memory to something like 8GB would make rofs >>> work. rofs is not very memory efficient - see >>> https://github.com/cloudius-systems/osv/issues/979. This is one of the >>> issues I would want to work next. >>> >>> >>> On Thursday, January 23, 2020 at 6:28:38 PM UTC-5, zhiting zhu wrote: >>>> >>>> The image is 2.5 G. >>>> >>>> On Thu, Jan 23, 2020 at 5:26 PM Waldek Kozaczuk <jwkoz...@gmail.com> >>>> wrote: >>>> >>>>> BTW how big is you image? >>>>> >>>>> I wounder if ROFS hangs because it runs out of memory when trying to >>>>> load files into memory. >>>>> >>>>> On Thursday, January 23, 2020 at 6:24:21 PM UTC-5, zhiting zhu wrote: >>>>>> >>>>>> Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I >>>>>> need to increase the qemu memory in upload_manifest.py otherwise it hangs >>>>>> on building zfs images. >>>>>> >>>>>> On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <jwkoz...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> It seems like it got stuck while trying to mount the filesystem. The >>>>>>> next boot message would normally be 'VFS: mounting devfs at /dev". >>>>>>> >>>>>>> I wonder if the image (usr.img) is somehow locked or something. Have >>>>>>> you tried to rebuild the image? Try zfs. >>>>>>> >>>>>>> Waldek >>>>>>> >>>>>>> On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote: >>>>>>>> >>>>>>>> native-example and python image works with qemu. It seems it only >>>>>>>> hangs on my custom tensorflow image. I'm only passing --verbose >>>>>>>> /python3 to >>>>>>>> run.py >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <jwkoz...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Are you passing any parameters to ./scripts/run.py? I would also >>>>>>>>> suggest to specify just single vCPU - it should make debugging easier. >>>>>>>>> >>>>>>>>> Also does it hang with this app only or others as well? Can you >>>>>>>>> try this: >>>>>>>>> ./scripts/build image=native-example >>>>>>>>> ./scripts/run.py >>>>>>>>> >>>>>>>>> Waldek >>>>>>>>> >>>>>>>>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the >>>>>>>>>> beginning. >>>>>>>>>> >>>>>>>>>> I'm seeing this: >>>>>>>>>> bsd: initializing - done >>>>>>>>>> VFS: mounting ramfs at / >>>>>>>>>> VFS: mounting devfs at /dev >>>>>>>>>> net: initializing - done >>>>>>>>>> vga: Add VGA device instance >>>>>>>>>> eth0: ethernet address: 52:54:00:12:34:56 >>>>>>>>>> virtio-blk: Add blk device instances 0 as vblk0, >>>>>>>>>> devsize=1192516096 >>>>>>>>>> random: virtio-rng registered as a source. >>>>>>>>>> random: intel drng, rdrand registered as a source. >>>>>>>>>> random: <Software, Yarrow> initialized >>>>>>>>>> VFS: unmounting /dev >>>>>>>>>> VFS: mounting rofs at /rofs >>>>>>>>>> random: device unblocked. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu < >>>>>>>>>> zhit...@cs.utexas.edu> wrote: >>>>>>>>>> >>>>>>>>>>> Here's the output log. See the file attached. >>>>>>>>>>> >>>>>>>>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <n...@scylladb.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk < >>>>>>>>>>>> jwkoz...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Can you send us the full output? I wonder if there are any >>>>>>>>>>>>> warnings before it? >>>>>>>>>>>>> >>>>>>>>>>>>> I see you are running this on firecracker. Can you run it >>>>>>>>>>>>> under qemu and connect to it with gdb and see if you get better >>>>>>>>>>>>> stacktrace? >>>>>>>>>>>>> >>>>>>>>>>>>> You can add another debug statement like that: >>>>>>>>>>>>> >>>>>>>>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>>>>>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>>>>>>>>>> ... >>>>>>>>>>>>> // Put breakpoint here >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> and try to see what statement causes the fault. Make sure to >>>>>>>>>>>>> do 'osv syms" to get as much debug info resolved as possible. (see >>>>>>>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I hope that "osv syms" will find the >>>>>>>>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it >>>>>>>>>>>> doesn't, maybe >>>>>>>>>>>> we can fix the order of when the array that "osv syms" uses gets >>>>>>>>>>>> written >>>>>>>>>>>> during loaded. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I wonder if this has to with the order of initializing the ELF >>>>>>>>>>>>> objects when called by dlopen(). >>>>>>>>>>>>> >>>>>>>>>>>>> Waldek >>>>>>>>>>>>> >>>>>>>>>>>>> PS. If no more clue next step would be to add an app so we can >>>>>>>>>>>>> build it and reproduce it? >>>>>>>>>>>>> >>>>>>>>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting >>>>>>>>>>>>> zhu wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm hitting this error: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT >>>>>>>>>>>>>> function >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing >>>>>>>>>>>>>> DT_INIT function >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>>>>>>>>>> DT_INIT_ARRAYSZ functions >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init >>>>>>>>>>>>>> func 0x1000388ad500 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init >>>>>>>>>>>>>> func 0x1000388ac8a0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init >>>>>>>>>>>>>> func 0x1000388ac8d0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init >>>>>>>>>>>>>> func 0x1000388ac900 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init >>>>>>>>>>>>>> func 0x1000388ac930 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init >>>>>>>>>>>>>> func 0x1000388ac960 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init >>>>>>>>>>>>>> func 0x1000388ac990 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init >>>>>>>>>>>>>> func 0x1000388ac9c0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init >>>>>>>>>>>>>> func 0x1000388ac9f0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init >>>>>>>>>>>>>> func 0x1000388aca20 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init >>>>>>>>>>>>>> func 0x1000388aca50 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init >>>>>>>>>>>>>> func 0x1000388aca80 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init >>>>>>>>>>>>>> func 0x1000388acab0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init >>>>>>>>>>>>>> func 0x1000388acae0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init >>>>>>>>>>>>>> func 0x1000388acb10 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init >>>>>>>>>>>>>> func 0x1000388acb40 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init >>>>>>>>>>>>>> func 0x1000388acb70 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init >>>>>>>>>>>>>> func 0x1000388acc50 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init >>>>>>>>>>>>>> func 0x1000388acc80 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init >>>>>>>>>>>>>> func 0x1000388accb0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init >>>>>>>>>>>>>> func 0x1000388acce0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init >>>>>>>>>>>>>> func 0x1000388acd10 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init >>>>>>>>>>>>>> func 0x1000388acd40 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init >>>>>>>>>>>>>> func 0x1000388acd70 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init >>>>>>>>>>>>>> func 0x1000388acda0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init >>>>>>>>>>>>>> func 0x1000388acdd0 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init >>>>>>>>>>>>>> func 0x1000388ace00 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init >>>>>>>>>>>>>> func 0x1000388ace50 >>>>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init >>>>>>>>>>>>>> func 0x1000388ace80 >>>>>>>>>>>>>> Aborted >>>>>>>>>>>>>> >>>>>>>>>>>>>> [backtrace] >>>>>>>>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>>>>>>>>>> exception_frame*)+59> >>>>>>>>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, >>>>>>>>>>>>>> int, exception_frame*)+26> >>>>>>>>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>>>>>>>>>> exception_frame*)+185> >>>>>>>>>>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>>>>>>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>>>>>>>>>> 0x0000000040341042 <elf::program::init_library(int, >>>>>>>>>>>>>> char**)+402> >>>>>>>>>>>>>> 0x00000000403473db >>>>>>>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>>>>>>>>>> std::vector<std::__cxx11::basic_string<char, >>>>>>>>>>>>>> std::char_traits<char>, >>>>>>>>>>>>>> std::allocator<char> >, >>>>>>>>>>>>>> std::allocator<std::__cxx11::basic_string<char, >>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>>>>>>>>>> 0x0000000040462f19 <dlopen+153> >>>>>>>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>>>>>>>>>> 0x006567617373656c <???+1936942444> >>>>>>>>>>>>>> 2020-01-22T17:13:51.345740567 >>>>>>>>>>>>>> [anonymous-instance:ERROR:vmm/src/lib.rs:1658] Failed to log >>>>>>>>>>>>>> metrics: Logger was not initialized. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is there any clue how to debug this? The function pointer >>>>>>>>>>>>>> seems to point to a valid address but I get a seg fault when >>>>>>>>>>>>>> executing it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Zhiting >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "OSv Development" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to osv...@googlegroups.com. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "OSv Development" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to osv...@googlegroups.com. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "OSv Development" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to osv...@googlegroups.com. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "OSv Development" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to osv...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OSv Development" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to osv...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "OSv Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to osv-dev+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/osv-dev/34332f2f-7a70-4711-bc34-ce19bccaedcd%40googlegroups.com >>> <https://groups.google.com/d/msgid/osv-dev/34332f2f-7a70-4711-bc34-ce19bccaedcd%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CA%2B3q14zjhSWU5%2Br4FB4qNV8P2Z2UidRiZDRYmPJovHwA3UUDdA%40mail.gmail.com.
