Yeah that is pretty big image given that the default memory size if 4GB. I
wonder if increasing the memory to something like 8GB would make rofs work.
rofs is not very memory efficient - see
https://github.com/cloudius-systems/osv/issues/979. This is one of the
issues I would want to work next.
On Thursday, January 23, 2020 at 6:28:38 PM UTC-5, zhiting zhu wrote:
>
> The image is 2.5 G.
>
> On Thu, Jan 23, 2020 at 5:26 PM Waldek Kozaczuk <jwkoz...@gmail.com
> <javascript:>> wrote:
>
>> BTW how big is you image?
>>
>> I wounder if ROFS hangs because it runs out of memory when trying to load
>> files into memory.
>>
>> On Thursday, January 23, 2020 at 6:24:21 PM UTC-5, zhiting zhu wrote:
>>>
>>> Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need
>>> to increase the qemu memory in upload_manifest.py otherwise it hangs on
>>> building zfs images.
>>>
>>> On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <jwkoz...@gmail.com>
>>> wrote:
>>>
>>>> It seems like it got stuck while trying to mount the filesystem. The
>>>> next boot message would normally be 'VFS: mounting devfs at /dev".
>>>>
>>>> I wonder if the image (usr.img) is somehow locked or something. Have
>>>> you tried to rebuild the image? Try zfs.
>>>>
>>>> Waldek
>>>>
>>>> On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote:
>>>>>
>>>>> native-example and python image works with qemu. It seems it only
>>>>> hangs on my custom tensorflow image. I'm only passing --verbose /python3
>>>>> to
>>>>> run.py
>>>>>
>>>>>
>>>>> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <jwkoz...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Are you passing any parameters to ./scripts/run.py? I would also
>>>>>> suggest to specify just single vCPU - it should make debugging easier.
>>>>>>
>>>>>> Also does it hang with this app only or others as well? Can you try
>>>>>> this:
>>>>>> ./scripts/build image=native-example
>>>>>> ./scripts/run.py
>>>>>>
>>>>>> Waldek
>>>>>>
>>>>>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote:
>>>>>>>
>>>>>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the
>>>>>>> beginning.
>>>>>>>
>>>>>>> I'm seeing this:
>>>>>>> bsd: initializing - done
>>>>>>> VFS: mounting ramfs at /
>>>>>>> VFS: mounting devfs at /dev
>>>>>>> net: initializing - done
>>>>>>> vga: Add VGA device instance
>>>>>>> eth0: ethernet address: 52:54:00:12:34:56
>>>>>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096
>>>>>>> random: virtio-rng registered as a source.
>>>>>>> random: intel drng, rdrand registered as a source.
>>>>>>> random: <Software, Yarrow> initialized
>>>>>>> VFS: unmounting /dev
>>>>>>> VFS: mounting rofs at /rofs
>>>>>>> random: device unblocked.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <zhit...@cs.utexas.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here's the output log. See the file attached.
>>>>>>>>
>>>>>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <n...@scylladb.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <
>>>>>>>>> jwkoz...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Can you send us the full output? I wonder if there are any
>>>>>>>>>> warnings before it?
>>>>>>>>>>
>>>>>>>>>> I see you are running this on firecracker. Can you run it under
>>>>>>>>>> qemu and connect to it with gdb and see if you get better stacktrace?
>>>>>>>>>>
>>>>>>>>>> You can add another debug statement like that:
>>>>>>>>>>
>>>>>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message.
>>>>>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) {
>>>>>>>>>> ...
>>>>>>>>>> // Put breakpoint here
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> and try to see what statement causes the fault. Make sure to do
>>>>>>>>>> 'osv syms" to get as much debug info resolved as possible. (see
>>>>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I hope that "osv syms" will find the
>>>>>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't,
>>>>>>>>> maybe
>>>>>>>>> we can fix the order of when the array that "osv syms" uses gets
>>>>>>>>> written
>>>>>>>>> during loaded.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I wonder if this has to with the order of initializing the ELF
>>>>>>>>>> objects when called by dlopen().
>>>>>>>>>>
>>>>>>>>>> Waldek
>>>>>>>>>>
>>>>>>>>>> PS. If no more clue next step would be to add an app so we can
>>>>>>>>>> build it and reproduce it?
>>>>>>>>>>
>>>>>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hey,
>>>>>>>>>>>
>>>>>>>>>>> I'm hitting this error:
>>>>>>>>>>>
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT
>>>>>>>>>>> function
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing
>>>>>>>>>>> DT_INIT function
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56
>>>>>>>>>>> DT_INIT_ARRAYSZ functions
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func
>>>>>>>>>>> 0x1000388ad500
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func
>>>>>>>>>>> 0x1000388ac8a0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func
>>>>>>>>>>> 0x1000388ac8d0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func
>>>>>>>>>>> 0x1000388ac900
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func
>>>>>>>>>>> 0x1000388ac930
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func
>>>>>>>>>>> 0x1000388ac960
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func
>>>>>>>>>>> 0x1000388ac990
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func
>>>>>>>>>>> 0x1000388ac9c0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func
>>>>>>>>>>> 0x1000388ac9f0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func
>>>>>>>>>>> 0x1000388aca20
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init
>>>>>>>>>>> func 0x1000388aca50
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init
>>>>>>>>>>> func 0x1000388aca80
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init
>>>>>>>>>>> func 0x1000388acab0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init
>>>>>>>>>>> func 0x1000388acae0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init
>>>>>>>>>>> func 0x1000388acb10
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init
>>>>>>>>>>> func 0x1000388acb40
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init
>>>>>>>>>>> func 0x1000388acb70
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init
>>>>>>>>>>> func 0x1000388acc50
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init
>>>>>>>>>>> func 0x1000388acc80
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init
>>>>>>>>>>> func 0x1000388accb0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init
>>>>>>>>>>> func 0x1000388acce0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init
>>>>>>>>>>> func 0x1000388acd10
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init
>>>>>>>>>>> func 0x1000388acd40
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init
>>>>>>>>>>> func 0x1000388acd70
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init
>>>>>>>>>>> func 0x1000388acda0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init
>>>>>>>>>>> func 0x1000388acdd0
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init
>>>>>>>>>>> func 0x1000388ace00
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init
>>>>>>>>>>> func 0x1000388ace50
>>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init
>>>>>>>>>>> func 0x1000388ace80
>>>>>>>>>>> Aborted
>>>>>>>>>>>
>>>>>>>>>>> [backtrace]
>>>>>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&,
>>>>>>>>>>> exception_frame*)+59>
>>>>>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int,
>>>>>>>>>>> exception_frame*)+26>
>>>>>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long,
>>>>>>>>>>> exception_frame*)+185>
>>>>>>>>>>> 0x000000004038f7b6 <page_fault+166>
>>>>>>>>>>> 0x000000004038e5f6 <???+1077470710>
>>>>>>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402>
>>>>>>>>>>> 0x00000000403473db
>>>>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char,
>>>>>>>>>>> std::char_traits<char>, std::allocator<char> >,
>>>>>>>>>>> std::vector<std::__cxx11::basic_string<char,
>>>>>>>>>>> std::char_traits<char>,
>>>>>>>>>>> std::allocator<char> >,
>>>>>>>>>>> std::allocator<std::__cxx11::basic_string<char,
>>>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715>
>>>>>>>>>>> 0x0000000040462f19 <dlopen+153>
>>>>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376>
>>>>>>>>>>> 0x006567617373656c <???+1936942444>
>>>>>>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/
>>>>>>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized.
>>>>>>>>>>>
>>>>>>>>>>> Is there any clue how to debug this? The function pointer seems
>>>>>>>>>>> to point to a valid address but I get a seg fault when executing
>>>>>>>>>>> it.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Zhiting
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "OSv Development" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to osv...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com
>>>>>>>>>>
>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "OSv Development" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to osv...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com
>>>>>>>>>
>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "OSv Development" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to osv...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "OSv Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to osv...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to osv...@googlegroups.com <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/34332f2f-7a70-4711-bc34-ce19bccaedcd%40googlegroups.com.
