Hi Waldek,
I am able to make the issue appear again by creating a big file (i.e. dd
if=/dev/urandom of=1GB.bin bs=64M count=16 iflag=fullblock), creating a new
app within the apps folder called big_file, setting the usr.manifest to
include /big_file/**: ${MODULE_DIR}/**, and running:
./scripts/build fs_size_mb=8192 image=python-from-host,java8,big_file
I updated the buf.h file, and the issue remains.
When I used fs=rofs instead, the issue returns but it doesn't appear until
I run the run.py script. Example: the last few lines of the build state:
First block: 5242316, blocks count: 5127
Directory entries count 43078
Symlinks count 10
Inodes count 43079
But when I run
./scripts/run.py -e "/python3.10"
I get the following message:
OSv v0.57.0-86-g873cb55a
Assertion failed: virt >= phys_mem (core/mmu.cc: virt_to_phys: 184)
[backtrace]
0x0000000040244a7c <__assert_fail+28>
0x00000000402b85fc <mmu::virt_to_phys(void*)+92>
0x00000000402ef3e2 <void mmu::virt_to_phys<virtio::vring::add_sg(void*,
unsigned int, virtio::vring_desc::flags)::{lambda(unsigned long, unsigned
long)#1}>(void*, unsigned long, virtio::vring::add_sg(void*, unsigned int,
virtio::vring_desc::flags)::{lambda(unsigned long, unsigned long)#1})+34>
0x00000000402ef148 <virtio::blk::make_request(bio*)+312>
0x00000000403bfd75 <multiplex_strategy+197>
0x00000000403cacbb <rofs_read_blocks(device*, unsigned long, unsigned long,
void*)+107>
0x00000000403c91fd <???+1077711357>
0x00000000403c1476 <sys_mount+694>
0x00000000403beadb <mount_rofs_rootfs+59>
0x0000000040239491 <do_main_thread(void*)+7809>
0x00000000403e4e69 <???+1077825129>
0x000000004037db2d <thread_main_c+45>
0x000000004030b361 <???+1076933473>
When I use Virtio-FS, the error does not seem to appear, but this seems to
cause different errors possibly. When trying to run Python in the
unikernel, I get:
sudo PATH=/usr/lib/qemu:$PATH ./scripts/run.py --virtio-fs-tag=myfs
--virtio-fs-dir=$(pwd)/build/export -e "/python3.10"
OSv v0.57.0-86-g873cb55a
eth0: 192.168.122.15
Booted up in 109.01 ms
Cmdline: /python3.10
Fatal Python error: _Py_HashRandomization_Init: failed to get random
numbers to initialize Python
Python runtime state: preinitialized
Hope this is enough to replicate the issue. Thank you!
On Monday, December 11, 2023 at 3:03:01 PM UTC-5 [email protected] wrote:
> Hi,
>
> On Thursday, December 7, 2023 at 6:43:00 PM UTC-5 Darren L wrote:
>
> Hi Waldek,
>
> Thanks for the quick response. For more details, I am trying to run a
> research prototype that requires both Python 3 and Java 8 to run correctly,
> and I placed the prototype's executable files (which includes large amounts
> of static data required to run the program) as an image in the apps
> directory to be linked to the image. The individual files are not huge (up
> to a few hundred MB each), but rather there are a lot of files that need to
> be included to run the prototype (totalling altogether 2-3 GB). I wasn't
> sure if this was the best approach. It seems that I could also dynamically
> link it in the .img file after the fact, since I only need the Python and
> Java images to run the program correctly?
>
> The error I am getting occurs when I run the following command:
> `./scripts/build
> fs_size_mb=8192 image=python-from-host,java8,prototype` where prototype
> contains my large executable files. The error I receive is during the build
> process. It will state:
>
> ```
> Assertion failed: virt >= phys_mem (core/mmu.cc: virt_to_phys: 184)
>
> [backtrace]
> 0x0000000040244a7c <__assert_fail+28>
> 0x00000000402b85fc <mmu::virt_to_phys(void*)+92>
> 0x00000000402ef3e2 <void mmu::virt_to_phys<virtio::vring::add_sg(void*,
> unsigned int, virtio::vring_desc::flags)::{lambda(unsigned long, unsigned
> long)#1}>(void*, unsigned long, virtio::vring::add_sg(void*, unsigned int,
> virtio::vring_desc::flags)::{lambda(unsigned long, unsigned long)#1})+34>
> 0x00000000402ef1e6 <virtio::blk::make_request(bio*)+470>
> 0x000010000006b1e5 <???+438757>
> 0x000010000009620a <???+614922>
> 0x000010000006e476 <???+451702>
> 0x000010000009620a <???+614922>
> 0x0000000040260e1e <???+1076235806>
> 0x0000000040260ef2 <taskqueue_thread_loop+82>
> 0x000000004037db2d <thread_main_c+45>
> 0x000000004030b361 <???+1076933473>
> ```
>
> I do not think this error has anything to do with the ELF layout or the
> kernel shift. I think this issue is different and it happens when OSv runs
> during the build process (ZFS builder) to create the ZFS disk and upload
> all files.
> I am still very interested in replicating and fixing it.
>
> My wild guess is that it may be caused by the bug somebody else discovered
> and fixed in his pull request -
> https://github.com/cloudius-systems/osv/pull/1284/files. Can you only
> update the include/osv/buf.h file to see if your problem goes away?
> Otherwise, I have to be able to replicate it somehow.
>
> Also, can you try to build a ROFS image (add fs=rofs to your build
> command) and run it?
>
> There is also an option to use Virtio-FS (see
> https://github.com/cloudius-systems/osv/wiki/virtio-fs).
>
> I'm using the most recent OSv pulled from Github. I'm not using node but
> Python. To my best understanding, this is what I got using readelf for Java
> and Python. I used the files within my own machine that were then
> transferred into OSv. I wasn't sure how to use `readelf` within the
> unikernel itself, so I might need guidance on using readelf within OSv if
> this is not sufficient.
>
> openjdk-8-zulu-full's java binary:
>
> ```
> Elf file type is EXEC (Executable file)
> Entry point 0x400570
> There are 9 program headers, starting at offset 64
>
>
> Program Headers:
> Type Offset VirtAddr PhysAddr FileSiz
> MemSiz Flg Align
> PHDR 0x000040 0x0000000000400040 0x0000000000400040 0x0001f8
> 0x0001f8 R E 0x8
> INTERP 0x000238 0x0000000000400238 0x0000000000400238 0x00001c
> 0x00001c R 0x1
> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
> LOAD 0x000000 0x0000000000400000 0x0000000000400000 0x0008c4
> 0x0008c4 R E 0x200000
> LOAD 0x000d98 0x0000000000600d98 0x0000000000600d98 0x00027c
> 0x000290 RW 0x200000
> DYNAMIC 0x000dd0 0x0000000000600dd0 0x0000000000600dd0 0x000210
> 0x000210 RW 0x8
> NOTE 0x000254 0x0000000000400254 0x0000000000400254 0x000044
> 0x000044 R 0x4
> GNU_EH_FRAME 0x000818 0x0000000000400818 0x0000000000400818 0x000024
> 0x000024 R 0x4
> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000
> 0x000000 RW 0x8
> GNU_RELRO 0x000d98 0x0000000000600d98 0x0000000000600d98 0x000268
> 0x000268 R 0x1
>
> Section to Segment mapping:
> Segment Sections...
> 00
> 01 .interp
> 02 .interp .note.ABI-tag .note.gnu.build-id .hash .gnu.hash .dynsym
> .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn .rela.plt
> .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame
> 03 .ctors .dtors .jcr .data.rel.ro .dynamic .got .got.plt .data
> .bss
> 04 .dynamic
> 05 .note.ABI-tag .note.gnu.build-id
> 06 .eh_frame_hdr
> 07
> 08 .ctors .dtors .jcr .data.rel.ro .dynamic .got
> ```
>
> python3.10.12:
>
> ```
> Elf file type is DYN (Position-Independent Executable file)
> Entry point 0x22cb80
>
> There are 13 program headers, starting at offset 64
>
> Program Headers:
> Type Offset VirtAddr PhysAddr FileSiz
> MemSiz Flg Align
> PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002d8
> 0x0002d8 R 0x8
> INTERP 0x000318 0x0000000000000318 0x0000000000000318 0x00001c
> 0x00001c R 0x1
> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
> LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x06c228
> 0x06c228 R 0x1000
> LOAD 0x06d000 0x000000000006d000 0x000000000006d000 0x2b0dad
> 0x2b0dad R E 0x1000
> LOAD 0x31e000 0x000000000031e000 0x000000000031e000 0x23ee58
> 0x23ee58 R 0x1000
> LOAD 0x55d810 0x000000000055e810 0x000000000055e810 0x045528
> 0x08b6c8 RW 0x1000
> DYNAMIC 0x562bc8 0x0000000000563bc8 0x0000000000563bc8 0x000220
> 0x000220 RW 0x8
>
> NOTE 0x000338 0x0000000000000338 0x0000000000000338 0x000030
> 0x000030 R 0x8
> NOTE 0x000368 0x0000000000000368 0x0000000000000368 0x000044
> 0x000044 R 0x4
> GNU_PROPERTY 0x000338 0x0000000000000338 0x0000000000000338 0x000030
> 0x000030 R 0x8
> GNU_EH_FRAME 0x4e72a4 0x00000000004e72a4 0x00000000004e72a4 0x012e74
> 0x012e74 R 0x4
>
> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000
> 0x000000 RW 0x10
> GNU_RELRO 0x55d810 0x000000000055e810 0x000000000055e810 0x0067f0
> 0x0067f0 R 0x1
>
> Section to Segment mapping:
> Segment Sections...
> 00
> 01 .interp
> 02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag
> .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt
> 03 .init .plt .plt.got .plt.sec .text .fini
> 04 .rodata .stapsdt.base .eh_frame_hdr .eh_frame
> 05 .init_array .fini_array .data.rel.ro .dynamic .got .data
> .PyRuntime .probes .bss
> 06 .dynamic
> 07 .note.gnu.property
> 08 .note.gnu.build-id .note.ABI-tag
> 09 .note.gnu.property
> 10 .eh_frame_hdr
> 11
> 12 .init_array .fini_array .data.rel.ro .dynamic .got
> ```
>
> Btw you can use a stock JDK or Python from your Linux host (see
> modules/openjdk9_1x-from-host -> image=...,openjdk9_1x-from-host) and the
> same for Python (apps/python-from-host -> image=...,python-from-host ).
>
> Let me know if you have any issues.
>
> Let me know if there is anything else you would like to see, or if you
> would like more clarification on anything mentioned above. Thank you for
> the help!
>
> Sincerely,
> Darren
>
> On Thursday, December 7, 2023 at 9:56:34 AM UTC-5 [email protected]
> wrote:
>
> Hi,
>
> The 2GB limit and the commit you are referring to should only limit the
> size of the position-dependent executables (these executables typically
> want to be loaded in a place where OSv kernel used to be before this
> commit).
>
> Are your executables larger than 2GB in size? Can you run 'readelf -W -l'
> against java and node like in this example:
>
> readelf -W -l /usr/lib/jvm/java-8-openjdk-amd64/bin/java
>
> Elf file type is DYN (Position-Independent Executable file)
> Entry point 0x10b0
> There are 13 program headers, starting at offset 64
>
> Program Headers:
> Type Offset VirtAddr PhysAddr FileSiz
> MemSiz Flg Align
> PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002d8
> 0x0002d8 R 0x8
> INTERP 0x000318 0x0000000000000318 0x0000000000000318 0x00001c
> 0x00001c R 0x1
> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
> LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000750
> 0x000750 R 0x1000
> LOAD 0x001000 0x0000000000001000 0x0000000000001000 0x0001a9
> 0x0001a9 R E 0x1000
> LOAD 0x002000 0x0000000000002000 0x0000000000002000 0x00011c
> 0x00011c R 0x1000
> LOAD 0x002d48 0x0000000000003d48 0x0000000000003d48 0x0002c8
> 0x0002d0 RW 0x1000
> DYNAMIC 0x002d58 0x0000000000003d58 0x0000000000003d58 0x000260
> 0x000260 RW 0x8
> NOTE 0x000338 0x0000000000000338 0x0000000000000338 0x000030
> 0x000030 R 0x8
> NOTE 0x000368 0x0000000000000368 0x0000000000000368 0x000044
> 0x000044 R 0x4
> GNU_PROPERTY 0x000338 0x0000000000000338 0x0000000000000338 0x000030
> 0x000030 R 0x8
> GNU_EH_FRAME 0x002038 0x0000000000002038 0x0000000000002038 0x000034
> 0x000034 R 0x4
> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000
> 0x000000 RW 0x10
> GNU_RELRO 0x002d48 0x0000000000003d48 0x0000000000003d48 0x0002b8
> 0x0002b8 R 0x1
>
> Can you give us more details about your use case and the error you are
> getting with the existing limit of 2GB (I am assuming there is a reason you
> want to increase it).
>
> Regards,
> Waldek
>
> On Thu, Dec 7, 2023 at 3:52 AM Darren Lim <[email protected]> wrote:
>
> Hello,
>
> Per the patch here (
> https://github.com/cloudius-systems/osv/commit/2a1795db8a22b0b963a64d068f5d8acc93e5785d),
>
> I was hoping to get help with making the changes to increase the kernel
> limit from 2GB to a larger size.
>
> For context, I am trying to load a large project (~3GB) into the
> unikernel, along with decently large languages (Java, Python) by creating a
> custom image for the build script. It currently complains in mmu.cc,
> stating:
>
> Assertion failed: virt >= phys_mem (core/mmu.cc: virt_to_phys: 184)
>
> Thank you!
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/a5c2844a-e03e-4ac4-8e0a-de81f575889fn%40googlegroups.com
>
> <https://groups.google.com/d/msgid/osv-dev/a5c2844a-e03e-4ac4-8e0a-de81f575889fn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/929419f2-2d51-461d-a13c-1443d72cdd2en%40googlegroups.com.
