The image is 2.5 G.
On Thu, Jan 23, 2020 at 5:26 PM Waldek Kozaczuk <jwkozac...@gmail.com>
wrote:
> BTW how big is you image?
>
> I wounder if ROFS hangs because it runs out of memory when trying to load
> files into memory.
>
> On Thursday, January 23, 2020 at 6:24:21 PM UTC-5, zhiting zhu wrote:
>>
>> Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need
>> to increase the qemu memory in upload_manifest.py otherwise it hangs on
>> building zfs images.
>>
>> On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <jwkoz...@gmail.com>
>> wrote:
>>
>>> It seems like it got stuck while trying to mount the filesystem. The
>>> next boot message would normally be 'VFS: mounting devfs at /dev".
>>>
>>> I wonder if the image (usr.img) is somehow locked or something. Have you
>>> tried to rebuild the image? Try zfs.
>>>
>>> Waldek
>>>
>>> On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote:
>>>>
>>>> native-example and python image works with qemu. It seems it only hangs
>>>> on my custom tensorflow image. I'm only passing --verbose /python3 to
>>>> run.py
>>>>
>>>>
>>>> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <jwkoz...@gmail.com>
>>>> wrote:
>>>>
>>>>> Are you passing any parameters to ./scripts/run.py? I would also
>>>>> suggest to specify just single vCPU - it should make debugging easier.
>>>>>
>>>>> Also does it hang with this app only or others as well? Can you try
>>>>> this:
>>>>> ./scripts/build image=native-example
>>>>> ./scripts/run.py
>>>>>
>>>>> Waldek
>>>>>
>>>>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote:
>>>>>>
>>>>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the
>>>>>> beginning.
>>>>>>
>>>>>> I'm seeing this:
>>>>>> bsd: initializing - done
>>>>>> VFS: mounting ramfs at /
>>>>>> VFS: mounting devfs at /dev
>>>>>> net: initializing - done
>>>>>> vga: Add VGA device instance
>>>>>> eth0: ethernet address: 52:54:00:12:34:56
>>>>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096
>>>>>> random: virtio-rng registered as a source.
>>>>>> random: intel drng, rdrand registered as a source.
>>>>>> random: <Software, Yarrow> initialized
>>>>>> VFS: unmounting /dev
>>>>>> VFS: mounting rofs at /rofs
>>>>>> random: device unblocked.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <zhit...@cs.utexas.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Here's the output log. See the file attached.
>>>>>>>
>>>>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <n...@scylladb.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <jwkoz...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Can you send us the full output? I wonder if there are any
>>>>>>>>> warnings before it?
>>>>>>>>>
>>>>>>>>> I see you are running this on firecracker. Can you run it under
>>>>>>>>> qemu and connect to it with gdb and see if you get better stacktrace?
>>>>>>>>>
>>>>>>>>> You can add another debug statement like that:
>>>>>>>>>
>>>>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message.
>>>>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) {
>>>>>>>>> ...
>>>>>>>>> // Put breakpoint here
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> and try to see what statement causes the fault. Make sure to do
>>>>>>>>> 'osv syms" to get as much debug info resolved as possible. (see
>>>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv).
>>>>>>>>>
>>>>>>>>
>>>>>>>> I hope that "osv syms" will find the
>>>>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't,
>>>>>>>> maybe
>>>>>>>> we can fix the order of when the array that "osv syms" uses gets
>>>>>>>> written
>>>>>>>> during loaded.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I wonder if this has to with the order of initializing the ELF
>>>>>>>>> objects when called by dlopen().
>>>>>>>>>
>>>>>>>>> Waldek
>>>>>>>>>
>>>>>>>>> PS. If no more clue next step would be to add an app so we can
>>>>>>>>> build it and reproduce it?
>>>>>>>>>
>>>>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey,
>>>>>>>>>>
>>>>>>>>>> I'm hitting this error:
>>>>>>>>>>
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT
>>>>>>>>>> function
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing
>>>>>>>>>> DT_INIT function
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56
>>>>>>>>>> DT_INIT_ARRAYSZ functions
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func
>>>>>>>>>> 0x1000388ad500
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func
>>>>>>>>>> 0x1000388ac8a0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func
>>>>>>>>>> 0x1000388ac8d0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func
>>>>>>>>>> 0x1000388ac900
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func
>>>>>>>>>> 0x1000388ac930
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func
>>>>>>>>>> 0x1000388ac960
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func
>>>>>>>>>> 0x1000388ac990
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func
>>>>>>>>>> 0x1000388ac9c0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func
>>>>>>>>>> 0x1000388ac9f0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func
>>>>>>>>>> 0x1000388aca20
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func
>>>>>>>>>> 0x1000388aca50
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func
>>>>>>>>>> 0x1000388aca80
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func
>>>>>>>>>> 0x1000388acab0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func
>>>>>>>>>> 0x1000388acae0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func
>>>>>>>>>> 0x1000388acb10
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func
>>>>>>>>>> 0x1000388acb40
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func
>>>>>>>>>> 0x1000388acb70
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func
>>>>>>>>>> 0x1000388acc50
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func
>>>>>>>>>> 0x1000388acc80
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func
>>>>>>>>>> 0x1000388accb0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func
>>>>>>>>>> 0x1000388acce0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func
>>>>>>>>>> 0x1000388acd10
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func
>>>>>>>>>> 0x1000388acd40
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func
>>>>>>>>>> 0x1000388acd70
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func
>>>>>>>>>> 0x1000388acda0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func
>>>>>>>>>> 0x1000388acdd0
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func
>>>>>>>>>> 0x1000388ace00
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func
>>>>>>>>>> 0x1000388ace50
>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_
>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func
>>>>>>>>>> 0x1000388ace80
>>>>>>>>>> Aborted
>>>>>>>>>>
>>>>>>>>>> [backtrace]
>>>>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&,
>>>>>>>>>> exception_frame*)+59>
>>>>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int,
>>>>>>>>>> exception_frame*)+26>
>>>>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long,
>>>>>>>>>> exception_frame*)+185>
>>>>>>>>>> 0x000000004038f7b6 <page_fault+166>
>>>>>>>>>> 0x000000004038e5f6 <???+1077470710>
>>>>>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402>
>>>>>>>>>> 0x00000000403473db
>>>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char,
>>>>>>>>>> std::char_traits<char>, std::allocator<char> >,
>>>>>>>>>> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>,
>>>>>>>>>> std::allocator<char> >,
>>>>>>>>>> std::allocator<std::__cxx11::basic_string<char,
>>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715>
>>>>>>>>>> 0x0000000040462f19 <dlopen+153>
>>>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376>
>>>>>>>>>> 0x006567617373656c <???+1936942444>
>>>>>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/
>>>>>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized.
>>>>>>>>>>
>>>>>>>>>> Is there any clue how to debug this? The function pointer seems
>>>>>>>>>> to point to a valid address but I get a seg fault when executing it.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Zhiting
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "OSv Development" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to osv...@googlegroups.com.
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com
>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "OSv Development" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to osv...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "OSv Development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to osv...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "OSv Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to osv...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com
>>> <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com
> <https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/CA%2B3q14wj%2BTXKdNrxH4PAG%3DbH%3D3uzxXW3NwNCuDVN57czk2xQZw%40mail.gmail.com.
