Hi,
Sorry for the late reply.
I did manage to replicate your issue. It turns out that when building so
large images, one needs more memory to run the so-called ZFS builder. After
increasing the memory from 512M to 1G I was able to build the image
successfully.
diff --git a/scripts/upload_manifest.py b/scripts/upload_manifest.py
index a3796f95..65e91a9c 100755
--- a/scripts/upload_manifest.py
+++ b/scripts/upload_manifest.py
@@ -164,7 +164,7 @@ def main():
console = '--console=serial'
zfs_builder_name = 'zfs_builder-stripped.elf'
- osv = subprocess.Popen('cd ../..; scripts/run.py -k --kernel-path
build/release/%s --arch=%s --vnc none -m 512 -c1 -i "%s"
--block-device-cache unsafe -s -e "%s --norandom --nomount --noinit
--preload-zfs-library /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/zfs/;
/zfs.so set compression=off osv" --forward tcp:127.0.0.1:%s-:10000' %
(zfs_builder_name,arch,image_path,console,upload_port), shell=True,
stdout=subprocess.PIPE)
+ osv = subprocess.Popen('cd ../..; scripts/run.py -k --kernel-path
build/release/%s --arch=%s --vnc none -m 1G -c1 -i "%s"
--block-device-cache unsafe -s -e "%s --norandom --nomount --noinit
--preload-zfs-library /tools/mkfs.so; /tools/cpiod.so --prefix /zfs/zfs/;
/zfs.so set compression=off osv" --forward tcp:127.0.0.1:%s-:10000' %
(zfs_builder_name,arch,image_path,console,upload_port), shell=True,
stdout=subprocess.PIPE)
upload(osv, manifest, depends, upload_port)
In case you see a similar error when running the image, also try to bump up
memory like - run.py -m 1G
We should probably detect this condition in core/mmu.cc accordingly and
better handle it possibly by informing user there is not enough physical
memory:
181
182 // For now, only allow non-mmaped areas. Later, we can either
183 // bounce such addresses, or lock them in memory and translate
184 assert(virt >= phys_mem);
185 return reinterpret_cast<uintptr_t>(virt) & (mem_area_size - 1);
186 }
On Wednesday, December 13, 2023 at 10:57:13 PM UTC-5 Darren L wrote:
> Hi Waldek,
>
> I am able to make the issue appear again by creating a big file (i.e. dd
> if=/dev/urandom of=1GB.bin bs=64M count=16 iflag=fullblock), creating a
> new app within the apps folder called big_file, setting the usr.manifest to
> include /big_file/**: ${MODULE_DIR}/**, and running:
>
> ./scripts/build fs_size_mb=8192 image=python-from-host,java8,big_file
>
> I updated the buf.h file, and the issue remains.
>
> When I used fs=rofs instead, the issue returns but it doesn't appear until
> I run the run.py script. Example: the last few lines of the build state:
>
> First block: 5242316, blocks count: 5127
> Directory entries count 43078
> Symlinks count 10
> Inodes count 43079
>
> But when I run
>
> ./scripts/run.py -e "/python3.10"
>
> I get the following message:
>
> OSv v0.57.0-86-g873cb55a
>
> Assertion failed: virt >= phys_mem (core/mmu.cc: virt_to_phys: 184)
>
> [backtrace]
> 0x0000000040244a7c <__assert_fail+28>
> 0x00000000402b85fc <mmu::virt_to_phys(void*)+92>
> 0x00000000402ef3e2 <void mmu::virt_to_phys<virtio::vring::add_sg(void*,
> unsigned int, virtio::vring_desc::flags)::{lambda(unsigned long, unsigned
> long)#1}>(void*, unsigned long, virtio::vring::add_sg(void*, unsigned int,
> virtio::vring_desc::flags)::{lambda(unsigned long, unsigned long)#1})+34>
> 0x00000000402ef148 <virtio::blk::make_request(bio*)+312>
> 0x00000000403bfd75 <multiplex_strategy+197>
> 0x00000000403cacbb <rofs_read_blocks(device*, unsigned long, unsigned
> long, void*)+107>
> 0x00000000403c91fd <???+1077711357>
> 0x00000000403c1476 <sys_mount+694>
> 0x00000000403beadb <mount_rofs_rootfs+59>
> 0x0000000040239491 <do_main_thread(void*)+7809>
> 0x00000000403e4e69 <???+1077825129>
> 0x000000004037db2d <thread_main_c+45>
> 0x000000004030b361 <???+1076933473>
>
> When I use Virtio-FS, the error does not seem to appear, but this seems to
> cause different errors possibly. When trying to run Python in the
> unikernel, I get:
>
> sudo PATH=/usr/lib/qemu:$PATH ./scripts/run.py --virtio-fs-tag=myfs
> --virtio-fs-dir=$(pwd)/build/export -e "/python3.10"
> OSv v0.57.0-86-g873cb55a
> eth0: 192.168.122.15
> Booted up in 109.01 ms
> Cmdline: /python3.10
> Fatal Python error: _Py_HashRandomization_Init: failed to get random
> numbers to initialize Python
> Python runtime state: preinitialized
>
> Hope this is enough to replicate the issue. Thank you!
>
> On Monday, December 11, 2023 at 3:03:01 PM UTC-5 jwkoz...@gmail.com wrote:
>
>> Hi,
>>
>> On Thursday, December 7, 2023 at 6:43:00 PM UTC-5 Darren L wrote:
>>
>> Hi Waldek,
>>
>> Thanks for the quick response. For more details, I am trying to run a
>> research prototype that requires both Python 3 and Java 8 to run correctly,
>> and I placed the prototype's executable files (which includes large amounts
>> of static data required to run the program) as an image in the apps
>> directory to be linked to the image. The individual files are not huge (up
>> to a few hundred MB each), but rather there are a lot of files that need to
>> be included to run the prototype (totalling altogether 2-3 GB). I wasn't
>> sure if this was the best approach. It seems that I could also dynamically
>> link it in the .img file after the fact, since I only need the Python and
>> Java images to run the program correctly?
>>
>> The error I am getting occurs when I run the following command:
>> `./scripts/build
>> fs_size_mb=8192 image=python-from-host,java8,prototype` where prototype
>> contains my large executable files. The error I receive is during the build
>> process. It will state:
>>
>> ```
>> Assertion failed: virt >= phys_mem (core/mmu.cc: virt_to_phys: 184)
>>
>> [backtrace]
>> 0x0000000040244a7c <__assert_fail+28>
>> 0x00000000402b85fc <mmu::virt_to_phys(void*)+92>
>> 0x00000000402ef3e2 <void mmu::virt_to_phys<virtio::vring::add_sg(void*,
>> unsigned int, virtio::vring_desc::flags)::{lambda(unsigned long, unsigned
>> long)#1}>(void*, unsigned long, virtio::vring::add_sg(void*, unsigned int,
>> virtio::vring_desc::flags)::{lambda(unsigned long, unsigned long)#1})+34>
>> 0x00000000402ef1e6 <virtio::blk::make_request(bio*)+470>
>> 0x000010000006b1e5 <???+438757>
>> 0x000010000009620a <???+614922>
>> 0x000010000006e476 <???+451702>
>> 0x000010000009620a <???+614922>
>> 0x0000000040260e1e <???+1076235806>
>> 0x0000000040260ef2 <taskqueue_thread_loop+82>
>> 0x000000004037db2d <thread_main_c+45>
>> 0x000000004030b361 <???+1076933473>
>> ```
>>
>> I do not think this error has anything to do with the ELF layout or the
>> kernel shift. I think this issue is different and it happens when OSv runs
>> during the build process (ZFS builder) to create the ZFS disk and upload
>> all files.
>> I am still very interested in replicating and fixing it.
>>
>> My wild guess is that it may be caused by the bug somebody else
>> discovered and fixed in his pull request -
>> https://github.com/cloudius-systems/osv/pull/1284/files. Can you only
>> update the include/osv/buf.h file to see if your problem goes away?
>> Otherwise, I have to be able to replicate it somehow.
>>
>> Also, can you try to build a ROFS image (add fs=rofs to your build
>> command) and run it?
>>
>> There is also an option to use Virtio-FS (see
>> https://github.com/cloudius-systems/osv/wiki/virtio-fs).
>>
>> I'm using the most recent OSv pulled from Github. I'm not using node but
>> Python. To my best understanding, this is what I got using readelf for Java
>> and Python. I used the files within my own machine that were then
>> transferred into OSv. I wasn't sure how to use `readelf` within the
>> unikernel itself, so I might need guidance on using readelf within OSv if
>> this is not sufficient.
>>
>> openjdk-8-zulu-full's java binary:
>>
>> ```
>> Elf file type is EXEC (Executable file)
>> Entry point 0x400570
>> There are 9 program headers, starting at offset 64
>>
>>
>> Program Headers:
>> Type Offset VirtAddr PhysAddr FileSiz
>> MemSiz Flg Align
>> PHDR 0x000040 0x0000000000400040 0x0000000000400040 0x0001f8
>> 0x0001f8 R E 0x8
>> INTERP 0x000238 0x0000000000400238 0x0000000000400238 0x00001c
>> 0x00001c R 0x1
>> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
>> LOAD 0x000000 0x0000000000400000 0x0000000000400000 0x0008c4
>> 0x0008c4 R E 0x200000
>> LOAD 0x000d98 0x0000000000600d98 0x0000000000600d98 0x00027c
>> 0x000290 RW 0x200000
>> DYNAMIC 0x000dd0 0x0000000000600dd0 0x0000000000600dd0 0x000210
>> 0x000210 RW 0x8
>> NOTE 0x000254 0x0000000000400254 0x0000000000400254 0x000044
>> 0x000044 R 0x4
>> GNU_EH_FRAME 0x000818 0x0000000000400818 0x0000000000400818 0x000024
>> 0x000024 R 0x4
>> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000
>> 0x000000 RW 0x8
>> GNU_RELRO 0x000d98 0x0000000000600d98 0x0000000000600d98 0x000268
>> 0x000268 R 0x1
>>
>> Section to Segment mapping:
>> Segment Sections...
>> 00
>> 01 .interp
>> 02 .interp .note.ABI-tag .note.gnu.build-id .hash .gnu.hash
>> .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn
>> .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame
>> 03 .ctors .dtors .jcr .data.rel.ro .dynamic .got .got.plt .data
>> .bss
>> 04 .dynamic
>> 05 .note.ABI-tag .note.gnu.build-id
>> 06 .eh_frame_hdr
>> 07
>> 08 .ctors .dtors .jcr .data.rel.ro .dynamic .got
>> ```
>>
>> python3.10.12:
>>
>> ```
>> Elf file type is DYN (Position-Independent Executable file)
>> Entry point 0x22cb80
>>
>> There are 13 program headers, starting at offset 64
>>
>> Program Headers:
>> Type Offset VirtAddr PhysAddr FileSiz
>> MemSiz Flg Align
>> PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002d8
>> 0x0002d8 R 0x8
>> INTERP 0x000318 0x0000000000000318 0x0000000000000318 0x00001c
>> 0x00001c R 0x1
>> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
>> LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x06c228
>> 0x06c228 R 0x1000
>> LOAD 0x06d000 0x000000000006d000 0x000000000006d000 0x2b0dad
>> 0x2b0dad R E 0x1000
>> LOAD 0x31e000 0x000000000031e000 0x000000000031e000 0x23ee58
>> 0x23ee58 R 0x1000
>> LOAD 0x55d810 0x000000000055e810 0x000000000055e810 0x045528
>> 0x08b6c8 RW 0x1000
>> DYNAMIC 0x562bc8 0x0000000000563bc8 0x0000000000563bc8 0x000220
>> 0x000220 RW 0x8
>>
>> NOTE 0x000338 0x0000000000000338 0x0000000000000338 0x000030
>> 0x000030 R 0x8
>> NOTE 0x000368 0x0000000000000368 0x0000000000000368 0x000044
>> 0x000044 R 0x4
>> GNU_PROPERTY 0x000338 0x0000000000000338 0x0000000000000338 0x000030
>> 0x000030 R 0x8
>> GNU_EH_FRAME 0x4e72a4 0x00000000004e72a4 0x00000000004e72a4 0x012e74
>> 0x012e74 R 0x4
>>
>> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000
>> 0x000000 RW 0x10
>> GNU_RELRO 0x55d810 0x000000000055e810 0x000000000055e810 0x0067f0
>> 0x0067f0 R 0x1
>>
>> Section to Segment mapping:
>> Segment Sections...
>> 00
>> 01 .interp
>> 02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag
>> .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt
>> 03 .init .plt .plt.got .plt.sec .text .fini
>> 04 .rodata .stapsdt.base .eh_frame_hdr .eh_frame
>> 05 .init_array .fini_array .data.rel.ro .dynamic .got .data
>> .PyRuntime .probes .bss
>> 06 .dynamic
>> 07 .note.gnu.property
>> 08 .note.gnu.build-id .note.ABI-tag
>> 09 .note.gnu.property
>> 10 .eh_frame_hdr
>> 11
>> 12 .init_array .fini_array .data.rel.ro .dynamic .got
>> ```
>>
>> Btw you can use a stock JDK or Python from your Linux host (see
>> modules/openjdk9_1x-from-host -> image=...,openjdk9_1x-from-host) and the
>> same for Python (apps/python-from-host -> image=...,python-from-host ).
>>
>> Let me know if you have any issues.
>>
>> Let me know if there is anything else you would like to see, or if you
>> would like more clarification on anything mentioned above. Thank you for
>> the help!
>>
>> Sincerely,
>> Darren
>>
>> On Thursday, December 7, 2023 at 9:56:34 AM UTC-5 jwkoz...@gmail.com
>> wrote:
>>
>> Hi,
>>
>> The 2GB limit and the commit you are referring to should only limit the
>> size of the position-dependent executables (these executables typically
>> want to be loaded in a place where OSv kernel used to be before this
>> commit).
>>
>> Are your executables larger than 2GB in size? Can you run 'readelf -W -l'
>> against java and node like in this example:
>>
>> readelf -W -l /usr/lib/jvm/java-8-openjdk-amd64/bin/java
>>
>> Elf file type is DYN (Position-Independent Executable file)
>> Entry point 0x10b0
>> There are 13 program headers, starting at offset 64
>>
>> Program Headers:
>> Type Offset VirtAddr PhysAddr FileSiz
>> MemSiz Flg Align
>> PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x0002d8
>> 0x0002d8 R 0x8
>> INTERP 0x000318 0x0000000000000318 0x0000000000000318 0x00001c
>> 0x00001c R 0x1
>> [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
>> LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000750
>> 0x000750 R 0x1000
>> LOAD 0x001000 0x0000000000001000 0x0000000000001000 0x0001a9
>> 0x0001a9 R E 0x1000
>> LOAD 0x002000 0x0000000000002000 0x0000000000002000 0x00011c
>> 0x00011c R 0x1000
>> LOAD 0x002d48 0x0000000000003d48 0x0000000000003d48 0x0002c8
>> 0x0002d0 RW 0x1000
>> DYNAMIC 0x002d58 0x0000000000003d58 0x0000000000003d58 0x000260
>> 0x000260 RW 0x8
>> NOTE 0x000338 0x0000000000000338 0x0000000000000338 0x000030
>> 0x000030 R 0x8
>> NOTE 0x000368 0x0000000000000368 0x0000000000000368 0x000044
>> 0x000044 R 0x4
>> GNU_PROPERTY 0x000338 0x0000000000000338 0x0000000000000338 0x000030
>> 0x000030 R 0x8
>> GNU_EH_FRAME 0x002038 0x0000000000002038 0x0000000000002038 0x000034
>> 0x000034 R 0x4
>> GNU_STACK 0x000000 0x0000000000000000 0x0000000000000000 0x000000
>> 0x000000 RW 0x10
>> GNU_RELRO 0x002d48 0x0000000000003d48 0x0000000000003d48 0x0002b8
>> 0x0002b8 R 0x1
>>
>> Can you give us more details about your use case and the error you are
>> getting with the existing limit of 2GB (I am assuming there is a reason you
>> want to increase it).
>>
>> Regards,
>> Waldek
>>
>> On Thu, Dec 7, 2023 at 3:52 AM Darren Lim <lucern...@gmail.com> wrote:
>>
>> Hello,
>>
>> Per the patch here (
>> https://github.com/cloudius-systems/osv/commit/2a1795db8a22b0b963a64d068f5d8acc93e5785d),
>>
>> I was hoping to get help with making the changes to increase the kernel
>> limit from 2GB to a larger size.
>>
>> For context, I am trying to load a large project (~3GB) into the
>> unikernel, along with decently large languages (Java, Python) by creating a
>> custom image for the build script. It currently complains in mmu.cc,
>> stating:
>>
>> Assertion failed: virt >= phys_mem (core/mmu.cc: virt_to_phys: 184)
>>
>> Thank you!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to osv-dev+u...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/osv-dev/a5c2844a-e03e-4ac4-8e0a-de81f575889fn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/osv-dev/a5c2844a-e03e-4ac4-8e0a-de81f575889fn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/3eb2890d-bc74-4a5e-a522-c66f9df9b03dn%40googlegroups.com.
