On 17.08.20 11:41, Peng Fan wrote: >> Subject: Re: [PATCH 06/10] Add libbaremetal > > Since we come to discuss in a bigger picture, how do you suggest > to proceed about the bootloader part? >
The key is to understand to use cases. What are your primary drivers to push the loader out of Linux? Is it "just" boot speed, ie. do you need a full-featured Jailhouse later on? Or how close would a static boot-time partitioning come to your use cases? Jan > Thanks, > Peng. > >> >> On 8/14/20 7:20 PM, Jan Kiszka wrote: >>> On 14.08.20 19:06, Ralf Ramsauer wrote: >>>> >>>> >>>> On 8/14/20 5:47 PM, Jan Kiszka wrote: >>>>> On 14.08.20 17:06, Ralf Ramsauer wrote: >>>>>> (+Cc Wolfgang, we recently had discussion on the boot process of >>>>>> Jailhouse) >>>>>> >>>>>> On 8/14/20 11:43 AM, Jan Kiszka wrote: >>>>>>> On 14.08.20 11:08, Peng Fan wrote: >>>>>>>>> Subject: Re: [PATCH 06/10] Add libbaremetal >>>>>>>>> >>>>>>>>> On 07.08.20 05:06, peng....@nxp.com wrote: >>>>>>>>>> From: Peng Fan <peng....@nxp.com> >>>>>>>>>> >>>>>>>>>> Add libbaremetal for sharing code between inmates and jailhouse >>>>>>>>>> baremetal loader. >>>>>>>>>> >>>>>>>>>> The Makefile code is copied from inmates, currently only >>>>>>>>>> string.c is moved from inmates to libbaremetal. In future, we >>>>>>>>>> might need to share uart/mmu and etc. >>>>>>>>> >>>>>>>>> Might quickly become confusing to have two libs. What prevents >>>>>>>>> renaming inmates/lib completely into libbaremetal? Sure, there >>>>>>>>> are some jailhouse specifics in inmates/lib, but those could >>>>>>>>> likely be put in some corner. >>>>>>>> >>>>>>>> How about rename inmates to baremetal? >>>>>>>> And add bootloader stuff under baremetal? >>>>>>> >>>>>>> We could do >>>>>>> >>>>>>> baremetal >>>>>> >>>>>> What we provide is a minimalist runtime environment, which is not >>>>>> just limited to baremetal applications (depends on the definition >>>>>> of baremetal, though). But… >>>>>> >>>>>>> ├── demo-inmates >>>>>>> ├── lib >>>>>>> ├── tests >>>>>>> └── tools >>>>>>> >>>>>>> and put the jailhouse loader under tools. Inmates is a "brand" in >>>>>>> Jailhouse context, so it should remain in some form. But it's >>>>>>> true, some >>>>>> >>>>>> … yep, the branding is the point why we should keep calling them >>>>>> 'inmates'. >>>>>> >>>>>>> of the existing demo inmates can already run without jailhouse. >>>>>> >>>>>> And that's the second point - we already have some kind of support >>>>>> for this kind of use case. However, I agree that I wouldn't expect >>>>>> the preloader inside inmates/. >>>>>> >>>>>>> >>>>>>> Ralf, what do you think? >>>>>> >>>>>> Peng, I just read the code of the loader and I try to understand >>>>>> what how it's exactly working. Nice work! Do I see that correctly: >>>>>> You basically "imitate" Linux in a way that you manually do the >>>>>> early setup (some register setup, GIC init, kick off all CPUs, hv stubs, >>>>>> ...). >>>>>> >>>>>> IOW, you bring the system to the same state, where the Linux driver >>>>>> would hand over Jailhouse. You activate Jailhouse, we then return >>>>>> to the loader which got lifted to the first cell, the root cell. >>>>>> Now you use the loader to partition the system, i.e., to create >>>>>> cells. Therefore, you use the hypercall interface, just like Linux >>>>>> would typically do it for us. Finally, there's the hand over to >>>>>> Linux. >>>>>> >>>>>> But do we really need a separate loader? Can't we just place the >>>>>> loader-related stuff at, say, the end of jailhouse.bin? >>>>>> >>>>>> To be more concrete, i think it should be possible to have the full >>>>>> preloader-thing located under hypervisor/arch/arm64/. It should >>>>>> also be possible to link everything that is required into >>>>>> jailhouse.bin - so we would get one universal binary instead of a >>>>>> chain of dependent binaries. [1] >>>>> >>>>> Interesting idea. >>>>> >>>>>> >>>>>> Pro: You can make early use of the UART in your preloader without >>>>>> having the need to, for example, duplicate drivers. That's quite >>>>>> useful for debugging if something goes wrong very early. Secondly, >>>>>> you can also piggyback on existing gic routines, no need to >>>>>> reimplement phys_processor_id, … >>>>>> >>>>>> The next thing after early boot would be the handover to jailhouse. >>>>>> Before the handover, we set a flag that indicates that we do an >>>>>> early boot of Jailhouse. We can use this flag to do the full >>>>>> partitioning of the system before we return to the guests in one >>>>>> single step. IOW, we never return to the preloader once the >>>>>> handover happened. We can do that, as u-boot already placed our >>>>>> guests and we just need to start them (as we would start regular >>>>>> cells). >>>>>> >>>>>> The amount of additional logic in the actual hypervisor core should >>>>>> be reasonably low. >>>>>> >>>>>> Did you already consider that strategy? Jan, what do you think? >>>>> >>>>> As we are already in visionary phase... There is another use case to >>>>> consider: >>>>> >>>>> I'd like to have a Jailhouse variant that comes without a root cell. >>>>> That means, we will need a booting procedure that creates all cells, >>>>> loads and starts them - and then drops all the logic that could >>>>> configure or destroy them (or never includes it in the first place - >>>>> build-time setting). This should reduce the Jailhouse runtime code >>>>> to much less than 10K LoC. >>>> >>>> Ok, that's indeed an awesome idea. >>>> >>>>> >>>>> Peng's approach is useful in case you do want to keep the >>>>> flexibility to reconfigure the system from a root cell, just like >>>>> you can do when >>>> >>>> To be honest, I consider the "dynamic" reconfigurability of Jailhouse >>>> as a development and testing utility rather than something that would >>>> ever be useful for operative usage. This includes, of course, >>>> jailhouse disabling. >>> >>> Reconfiguration, yes, but reloading can be a topic in certain scenarios. >>> The current config locking mechanisms allows to keep this door for the >>> root cell open, even while there is a critical cell running. >>> >>>> >>>> What would be great would be a single-shot 'jailhouse freeze' (or, >>>> ha, 'jailhouse detention') call, that throws away the whole hypercall >>>> interface. Having such a thing, we an drop everything that is somehow >>>> related to cell management: create, start, stop, destroy, stats, >>>> disable, enable, ... >>>> >>>> And that would in fact be no big deal: Annotate all affected >>>> routines, replace hyp vectors with stubs, drop the section, done. >>>> Every code that has no other users than from a hypercall can be >>>> dropped. Or do I miss something? >>> >>> I haven't thought this through in details yet, but I would be >>> surprised if there weren't at least some smaller challenges in >>> implementing that cleanly. Still, it's mostly about disabling code. >>> >>>> >>>>> booting via Linux. The idea of adding a preloader is minimal >>>>> invasive to the existing setup. My root-less mode will be maximal >>>>> invasive, >> though. >>>> >>>> Why do you think that it would be maximal invasive? >>> >>> As it changes the current hypervisor code for sure. Just loading >>> Jailhouse via a different thing than Linux does not bring many >>> changes, at least to the core. See this series. >>> >>>> >>>> Oh btw: who would receive the freed memory? Would it remain in the >>>> hypervisor or be assigned back to a cell? Though that should probably >>>> only be a few pages. >>> >>> What freed memory? When things are only created, nothing will be freed >>> anymore. There will also be no jailhouse disable. >> >> There will, e.g., be a cell_create routine, that will be called once per >> cell. After >> the creation of those cells, drop it, we won't need it again. >> >> If we have a universal binary, there will be a jailhouse disable will be >> present >> in the beginning. Once we decide freeze the configuration, drop it. If we >> make >> those things compile-time configurable, then it has potential to become a >> variant hell. >> >>> >>>> >>>>> If root-less would be sufficient for cases you do not want >>>>> Linux-based boot, we can skip this approach and head for root-less full >> steam. >>>> >>>> root-less means that the detention call (I like that) comes after >>>> cell creation, but before cell start. .oO(we can even drop the >>>> cell_start code, if we find a smart strategy to drop code right >>>> before vmreturn) >>>> >>> >>> I would rather envision some alternative configuration that contains >>> all cell configs, not just the root cell (which would become a >>> non-root cell as well). Loading would have to happen into the target >>> memory regions before jailhouse enable, and the enabling would also >>> imply starting all cells. >>> >>>>> >>>>> If there are use cases for all three variant, we can try to look for >>>>> common pieces in the two Linux-free options. My root-less mode would >>>>> also need a single-step "create and start all cells", e.g. And your >>>>> idea to include the bootstrap logic into an init section of >>>>> jailhouse.bin is also attractive for both. >>>> >>>> Yes, I see the overlaps. By the way... Why don't we actually do: >>>> >>>> $ cat jh.bin sys.cell c1.cell c2.cell > jh.image >> >> Even better: >> >> cat preloader.bin jh.bin sys.cell c1.cell c2.cell > jh.image >> >> Again, the preloader.bin is still a binary image that only contains the >> preloader >> section, but u-boot can directly jump into it. It is still developed under >> hypervisor/arch/foo and hence has full visibility of all symbols that are >> available in regular jailhouse code and can potentially use them. We just >> exploit the linker to export that section to a separate binary. >> >> If someone wants bare-metal boot mode, we just glue things together. >> >>>> >>>> for the bare metal case? >>> >>> That comes close to what I have in mind, config-wise. sys.cell would >>> still need a counter of cells that follow (simple to add to struct >>> jailhouse_system). >> >> Not necessarily. We can also define a zero-termination. Once the first byte >> of >> the JHCELL magic isn't present, we're done. But that's details. >> >> Ralf >> >>> >>> Jan >>> -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/4bfa6a8a-e4a8-9b4b-eb10-820242fab80a%40siemens.com.
