> Subject: Re: [PATCH 06/10] Add libbaremetal > > On 17.08.20 11:41, Peng Fan wrote: > >> Subject: Re: [PATCH 06/10] Add libbaremetal > > > > Since we come to discuss in a bigger picture, how do you suggest to > > proceed about the bootloader part? > > > > The key is to understand to use cases. What are your primary drivers to push > the loader out of Linux?
We have product using aarch64 jailhouse with aarch32 root cell Linux. With Linux as the loader, we have to add lots of hacks. Alought we are trying to upstream aarch64 jailhouse with aarch32 inmate cell, we consider to make root cell in aarch32 mode finally. Is it "just" boot speed, ie. do you need a full-featured > Jailhouse later on? Need a full-featured jailhouse. Or how close would a static boot-time partitioning come to > your use cases? Not evaluated this. But if it could support communication between cells, it should be fine to use it. Thanks, Peng. > > Jan > > > Thanks, > > Peng. > > > >> > >> On 8/14/20 7:20 PM, Jan Kiszka wrote: > >>> On 14.08.20 19:06, Ralf Ramsauer wrote: > >>>> > >>>> > >>>> On 8/14/20 5:47 PM, Jan Kiszka wrote: > >>>>> On 14.08.20 17:06, Ralf Ramsauer wrote: > >>>>>> (+Cc Wolfgang, we recently had discussion on the boot process of > >>>>>> Jailhouse) > >>>>>> > >>>>>> On 8/14/20 11:43 AM, Jan Kiszka wrote: > >>>>>>> On 14.08.20 11:08, Peng Fan wrote: > >>>>>>>>> Subject: Re: [PATCH 06/10] Add libbaremetal > >>>>>>>>> > >>>>>>>>> On 07.08.20 05:06, peng....@nxp.com wrote: > >>>>>>>>>> From: Peng Fan <peng....@nxp.com> > >>>>>>>>>> > >>>>>>>>>> Add libbaremetal for sharing code between inmates and > >>>>>>>>>> jailhouse baremetal loader. > >>>>>>>>>> > >>>>>>>>>> The Makefile code is copied from inmates, currently only > >>>>>>>>>> string.c is moved from inmates to libbaremetal. In future, we > >>>>>>>>>> might need to share uart/mmu and etc. > >>>>>>>>> > >>>>>>>>> Might quickly become confusing to have two libs. What prevents > >>>>>>>>> renaming inmates/lib completely into libbaremetal? Sure, there > >>>>>>>>> are some jailhouse specifics in inmates/lib, but those could > >>>>>>>>> likely be put in some corner. > >>>>>>>> > >>>>>>>> How about rename inmates to baremetal? > >>>>>>>> And add bootloader stuff under baremetal? > >>>>>>> > >>>>>>> We could do > >>>>>>> > >>>>>>> baremetal > >>>>>> > >>>>>> What we provide is a minimalist runtime environment, which is not > >>>>>> just limited to baremetal applications (depends on the definition > >>>>>> of baremetal, though). But… > >>>>>> > >>>>>>> ├── demo-inmates > >>>>>>> ├── lib > >>>>>>> ├── tests > >>>>>>> └── tools > >>>>>>> > >>>>>>> and put the jailhouse loader under tools. Inmates is a "brand" > >>>>>>> in Jailhouse context, so it should remain in some form. But it's > >>>>>>> true, some > >>>>>> > >>>>>> … yep, the branding is the point why we should keep calling them > >>>>>> 'inmates'. > >>>>>> > >>>>>>> of the existing demo inmates can already run without jailhouse. > >>>>>> > >>>>>> And that's the second point - we already have some kind of > >>>>>> support for this kind of use case. However, I agree that I > >>>>>> wouldn't expect the preloader inside inmates/. > >>>>>> > >>>>>>> > >>>>>>> Ralf, what do you think? > >>>>>> > >>>>>> Peng, I just read the code of the loader and I try to understand > >>>>>> what how it's exactly working. Nice work! Do I see that correctly: > >>>>>> You basically "imitate" Linux in a way that you manually do the > >>>>>> early setup (some register setup, GIC init, kick off all CPUs, hv > stubs, ...). > >>>>>> > >>>>>> IOW, you bring the system to the same state, where the Linux > >>>>>> driver would hand over Jailhouse. You activate Jailhouse, we then > >>>>>> return to the loader which got lifted to the first cell, the root cell. > >>>>>> Now you use the loader to partition the system, i.e., to create > >>>>>> cells. Therefore, you use the hypercall interface, just like > >>>>>> Linux would typically do it for us. Finally, there's the hand > >>>>>> over to Linux. > >>>>>> > >>>>>> But do we really need a separate loader? Can't we just place the > >>>>>> loader-related stuff at, say, the end of jailhouse.bin? > >>>>>> > >>>>>> To be more concrete, i think it should be possible to have the > >>>>>> full preloader-thing located under hypervisor/arch/arm64/. It > >>>>>> should also be possible to link everything that is required into > >>>>>> jailhouse.bin - so we would get one universal binary instead of a > >>>>>> chain of dependent binaries. [1] > >>>>> > >>>>> Interesting idea. > >>>>> > >>>>>> > >>>>>> Pro: You can make early use of the UART in your preloader without > >>>>>> having the need to, for example, duplicate drivers. That's quite > >>>>>> useful for debugging if something goes wrong very early. > >>>>>> Secondly, you can also piggyback on existing gic routines, no > >>>>>> need to reimplement phys_processor_id, … > >>>>>> > >>>>>> The next thing after early boot would be the handover to jailhouse. > >>>>>> Before the handover, we set a flag that indicates that we do an > >>>>>> early boot of Jailhouse. We can use this flag to do the full > >>>>>> partitioning of the system before we return to the guests in one > >>>>>> single step. IOW, we never return to the preloader once the > >>>>>> handover happened. We can do that, as u-boot already placed our > >>>>>> guests and we just need to start them (as we would start regular > >>>>>> cells). > >>>>>> > >>>>>> The amount of additional logic in the actual hypervisor core > >>>>>> should be reasonably low. > >>>>>> > >>>>>> Did you already consider that strategy? Jan, what do you think? > >>>>> > >>>>> As we are already in visionary phase... There is another use case > >>>>> to > >>>>> consider: > >>>>> > >>>>> I'd like to have a Jailhouse variant that comes without a root cell. > >>>>> That means, we will need a booting procedure that creates all > >>>>> cells, loads and starts them - and then drops all the logic that > >>>>> could configure or destroy them (or never includes it in the first > >>>>> place - build-time setting). This should reduce the Jailhouse > >>>>> runtime code to much less than 10K LoC. > >>>> > >>>> Ok, that's indeed an awesome idea. > >>>> > >>>>> > >>>>> Peng's approach is useful in case you do want to keep the > >>>>> flexibility to reconfigure the system from a root cell, just like > >>>>> you can do when > >>>> > >>>> To be honest, I consider the "dynamic" reconfigurability of > >>>> Jailhouse as a development and testing utility rather than > >>>> something that would ever be useful for operative usage. This > >>>> includes, of course, jailhouse disabling. > >>> > >>> Reconfiguration, yes, but reloading can be a topic in certain scenarios. > >>> The current config locking mechanisms allows to keep this door for > >>> the root cell open, even while there is a critical cell running. > >>> > >>>> > >>>> What would be great would be a single-shot 'jailhouse freeze' (or, > >>>> ha, 'jailhouse detention') call, that throws away the whole > >>>> hypercall interface. Having such a thing, we an drop everything > >>>> that is somehow related to cell management: create, start, stop, > >>>> destroy, stats, disable, enable, ... > >>>> > >>>> And that would in fact be no big deal: Annotate all affected > >>>> routines, replace hyp vectors with stubs, drop the section, done. > >>>> Every code that has no other users than from a hypercall can be > >>>> dropped. Or do I miss something? > >>> > >>> I haven't thought this through in details yet, but I would be > >>> surprised if there weren't at least some smaller challenges in > >>> implementing that cleanly. Still, it's mostly about disabling code. > >>> > >>>> > >>>>> booting via Linux. The idea of adding a preloader is minimal > >>>>> invasive to the existing setup. My root-less mode will be maximal > >>>>> invasive, > >> though. > >>>> > >>>> Why do you think that it would be maximal invasive? > >>> > >>> As it changes the current hypervisor code for sure. Just loading > >>> Jailhouse via a different thing than Linux does not bring many > >>> changes, at least to the core. See this series. > >>> > >>>> > >>>> Oh btw: who would receive the freed memory? Would it remain in the > >>>> hypervisor or be assigned back to a cell? Though that should > >>>> probably only be a few pages. > >>> > >>> What freed memory? When things are only created, nothing will be > >>> freed anymore. There will also be no jailhouse disable. > >> > >> There will, e.g., be a cell_create routine, that will be called once > >> per cell. After the creation of those cells, drop it, we won't need it > >> again. > >> > >> If we have a universal binary, there will be a jailhouse disable will > >> be present in the beginning. Once we decide freeze the configuration, > >> drop it. If we make those things compile-time configurable, then it > >> has potential to become a variant hell. > >> > >>> > >>>> > >>>>> If root-less would be sufficient for cases you do not want > >>>>> Linux-based boot, we can skip this approach and head for root-less > >>>>> full > >> steam. > >>>> > >>>> root-less means that the detention call (I like that) comes after > >>>> cell creation, but before cell start. .oO(we can even drop the > >>>> cell_start code, if we find a smart strategy to drop code right > >>>> before vmreturn) > >>>> > >>> > >>> I would rather envision some alternative configuration that contains > >>> all cell configs, not just the root cell (which would become a > >>> non-root cell as well). Loading would have to happen into the target > >>> memory regions before jailhouse enable, and the enabling would also > >>> imply starting all cells. > >>> > >>>>> > >>>>> If there are use cases for all three variant, we can try to look > >>>>> for common pieces in the two Linux-free options. My root-less mode > >>>>> would also need a single-step "create and start all cells", e.g. > >>>>> And your idea to include the bootstrap logic into an init section > >>>>> of jailhouse.bin is also attractive for both. > >>>> > >>>> Yes, I see the overlaps. By the way... Why don't we actually do: > >>>> > >>>> $ cat jh.bin sys.cell c1.cell c2.cell > jh.image > >> > >> Even better: > >> > >> cat preloader.bin jh.bin sys.cell c1.cell c2.cell > jh.image > >> > >> Again, the preloader.bin is still a binary image that only contains > >> the preloader section, but u-boot can directly jump into it. It is > >> still developed under hypervisor/arch/foo and hence has full > >> visibility of all symbols that are available in regular jailhouse > >> code and can potentially use them. We just exploit the linker to export > >> that > section to a separate binary. > >> > >> If someone wants bare-metal boot mode, we just glue things together. > >> > >>>> > >>>> for the bare metal case? > >>> > >>> That comes close to what I have in mind, config-wise. sys.cell would > >>> still need a counter of cells that follow (simple to add to struct > >>> jailhouse_system). > >> > >> Not necessarily. We can also define a zero-termination. Once the > >> first byte of the JHCELL magic isn't present, we're done. But that's > >> details. > >> > >> Ralf > >> > >>> > >>> Jan > >>> > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate > Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/DB6PR0402MB276074B8DEA27776C106C72D885C0%40DB6PR0402MB2760.eurprd04.prod.outlook.com.
