On Wed, Feb 13, 2019 at 7:03 AM Waldek Kozaczuk <jwkozac...@gmail.com> wrote:
> As you know OSv supports running unmodified Linux executable as long as > they use subset of Linux API implemented by OSv. On other hand OSv kernel > is not compatible with Linux as far as boot process goes. By that I mean > one cannot boot OSv on hypervisors that support booting some form of Linux > kernel - qemu with --kernel option, hyperkit with -kexec option and > firecracker. The Linux boot protocol is quite involved and carries a lot of > legacy so I advocate we only implement the minimal bits of it we need to > run OSv on 3 mentioned hypervisors. > > I also think it would be desirable to adhere to the following constraints: > > - keep minimum number of boot artifacts (usr.img, boot.bin, > lzloader.elf, loader.elf, etc) > > Yes, although I am not sure it will ever be possible to end up with just one target to be both a "disk image" (usr.img) and "something bootable with --kernel". I am not familiar enough with the details to say anything smart about this, you'll just have to investigate the details yourself (as it seems you've been doing). The only thing that would somewhat annoy me is that if we keep many copies of large files with tiny differences between them - in which case it would be nice to consider whether all of them really need to exist. > > - keep OSv boot process and artifacts backwards compatible with 16-bit > mode on Xen, QEMU, VirtualBox, VMware (this really means that usr.img needs > to stay bootable on those platforms but artifacts like lzloader.elf or > loader.elf do not need to stay the same) > > Let me recap how OSv process works now (for details please look at these > somewhat outdated Wiki pages - > https://github.com/cloudius-systems/osv/wiki/OSv-early-boot-(MBR) and > https://github.com/cloudius-systems/osv/wiki/OSv-lzloader-and-early-loader > ). > If you create an updated wiki page with all the (up to date) details, that would be great :-) > The OSv boot process relies on following artifacts produced by build > process: > > - loader.elf - 64-bit kernel ELF file with 32-bit entry point start32 > - get placed at 0x200000 in memory by uncompress_kernel > - start32 does following: > - sets up GDT for 32-bit and 64-bit > - sets up and enables paging > - eventually switches to long mode (64-bit) by jumping to start64 > - lzloader.elf - 32-bit ELF file that probably does not need to be > an ELF (*somewhat similar to Linux'es vmlinuz file*?) > - it is loaded by logic in boot.bin (boot16.S) from disk and placed > at 0x100000 in memory > - it is comprised of compressed loader.elf and fastlz code to > decompress it chunk-by-chunk in "almost-in-place" fashion so it ends up > at > the address 0x200000 in memory > - boot.bin - 512-bytes long 16-/32-bit binary produced from boot16.S > that: > - loads lzloader.elf from disk in 32K chunks at 0x100000 in memory > - loads command line from disk into memory (part of multiboot info > struct) > - detects available memory and places info in multiboot info struct > - switches to protected mode and calls uncompress_kernel (code in > lzloader.elf) > - finally calls start32 (code in loader.elf) > - usr.img that is comprised of boot.bin, cmdline and lzloader.elf and > looks like this: > - first 512 bytes are boot.bin > - > - it is MBR (Master Boot Record) > - from 513 until 64K-1 there is a command line followed with 0-s > - at 64K offset (128 blocks of 512 bytes) starts lzloader.elf > - > - 0......512.................65536=64k..... > - [boot.bin][cmdline]00000000[lzloader.elf] > > > The Linux boot protocol is documented here - > https://www.kernel.org/doc/Documentation/x86/boot.txt and I think we only > care about "32-bit BOOT PROTOCOL" of it (required by qemu --kernel and > hyperkit -kexec) and some subset of "64-bit BOOT PROTOCOL" (required by > firecracker).\ > > In order to make OSv comply with either of these two Linux boot flavors, > two aspects of it need to be considered: > > 1. The kernel loader file layout and what information goes in there to > satisfy particular hypervisor when loading the kernel file. > - in case of "32-bit BOOT PROTOCOL" the file is expected to be in > "vmlinuz" format and have so called "zero page" information (struct > boot_params) located at 0x1f1 offset (see > > https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/bootparam.h > ) > > So can we change lzloader.elf (the "not really elf" file) to be in this vmlinuz format? Would we want to? > > - in case of firecracker the loader file needs to be a 64-bit ELF with > 64-bit entry point - almost like OSv loader.elf except loader.elf has > 32-bit entry point > > For Linux, the same file can serve both firecracker and and "--kernel"? How? > > 1. The information that is passed by hypervisor to the Linux guest > (command line and memory info) and the VM state hypervisor sets before > calling kernel entry point. > - the command line and memory info gets set in proper fields of > boot_params struct field; the boot_params location in memory is > specified > in esi/rdi register > - in case of "32-bit BOOT PROTOCOL" the VM is set to 32-bit mode > and in general its states resembles roughly what start32 in OSv expects > I > think > - in case of firecracker the VM is set to 64-bit mode with GDT and > page tables setup Linux-way (please see this diff - > > https://github.com/wkozaczuk/osv/commit/3628f5607ae9dee8d812456571360f32e3c1aa29#diff-ee660a0822ff34de6d00ee9b0c382ce8- > to see changes I had to make - just a hack not a final solution - to > make > OSv loader.elf to boot on firecracker) > > So given all that I was wondering what are you thoughts/ideas of making > OSv be Linux-boot compliant enough so it boots on firecracker, QEMU > --kernel and hyperkit -exec? > It seems you have good ideas on how to do this - better than I have. > Here are my rough ideas: > > 1. Convert lzloader.elf to lzloader.bin that has enough of vmlinuz > format (zero page). > > Sounds good. > > 1. Optionally introduce not-compressed version of lzloader.bin. > > Sounds good because one of the purposes of the compression was to make loading from disk faster when the 16-bit loading is slow. With "-kernel", this optimization may be counter-productive. Maybe. > > 1. Somehow enhance loader.elf to support both 32-bit and 64-bit boot. > This seems impossible as ELF can only have one entry point. > - Maybe move protected mode setup logic from boot.S to lzloader.bin > and make start64 the entry point of loader.elf? > - Or maybe make lzloader.bin jump to start32 but make start32 no > longer an entry point of loader.elf? > > Finally in case of firecracker which set VM to long mode and sets up > paging the Linux way, what would be the proper way of resetting the VM > state (paging) to OSv way? Switching back to 32-bit mode and calling > start32? > > Waldek > > -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to osv-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
