On Thu, 6 Apr 2017, Volodymyr Babchuk wrote: > Hello all, > > I want to discuss EL0 (native) applications for XEN. This will be relatively > long e-mail with requirements, proposed design and my PoC results. > > So, why we want XEN native applications in the first place? I see the > following > reasons: > > 1. Isolation. I see XEN as a sort of micro-kernel, so there are no place for > device drivers, emulators, specific SMC handlers, hypervisor extension, etc.. > > 2. Modularity. Just look at Linux kernel. Obviously, for different devices we > can load different drivers. > > 3. Performance. Native application should be faster than stub domain, or there > will be no sense in it. > > 4. Ease of use. I want to make call to EL0 app as easy as possible. > Ideally - as a function call. > > Actually, no one wants extra code in hypervisor, so reasons (1) and (2) are > most > important. I know that there was tries to do such thing in x86 but with > different approach. I want describe my idea for arm64. > > Native application is an another domain type. It has own vCPU (only one at > this > moment) Native app is loaded as any other kernel, using ELF loader. > It looks like another stub-domain such as MiniOS, but there are two big > differences:
Could you describe the reason why you are suggesting it? Unless strictly necessary, I wouldn't go down the vcpu route, because as soon as we bring a vcpu into the picture, we have a number of problems, including scheduling, affinity, etc. It is also user-visible (xl vcpu-list) which I don't think it should be. I understand that one of the goals is "Modularity", which makes us think of an ELF loader, such as the one for a new domain. I agree that modularity is important, but I would solve it as a second step. In first instance, I would limit the scope to run some code under /xen/arch/arm/apps or, better, /apps (for example) in a lower privilege mode. After that is done and working, I am sure we can find a way to dynamically load more apps at run time. > 1. MiniOS has event loop that serves requests from hypervisor. Native > application does not has such loop. It has one entry point where you jump > every > time when you need something from it. > > 2. Native application runs in EL0 mode, so it does not have access to MMU, > it can't handle vIQRs, exceptions and so on. XEN does all this for it. > > You can find example native application at [1]. I used exactly this one to > benchmark my implementation. Mostly it is inspired by approach used in TEE. > Actually, I took some code directly from OP-TEE Trusted Application library. > In app_entry.c you can find entry point - __app_entry(). It takes function > number and some parameters that will be passed to a function. I probably going > to change ABI a bit, but basic idea will be the same. > > Function number will be something like APP_INIT, APP_HANDLE_SMC > or APP_HANDLE_MMIO... I think you got the idea. I also implemented two > syscalls > (via old plain SVC instruction). app_log() writes to XEN log, app_return() > exits > from application back to hypervisor. We will need other syscalls like > app_call_smc(), app_map_guest_page(), app_map_io(), etc. > > Now, back to XEN. Classic way to handle something with stubdomain is to > write request to a ring buffer, fire an event through event channel, that will > trigger vIRQ in stubdomain and stubdomain's vCPU will be scheduled to handle > a request. Problem it that you can't control scheduler, so you don't know > when your request will be really handled, which in not fine in some > embedded use cases. > > There is how I see handling requests with native application: > > 0. Hypervisor pauses requester vCPU > 1. Hypervisor either passes parameters via registers or writes request to a > shared page/ring buffer. > 2. Then in sets PC of native app vCPU to entry point and initializes r0-r7 > with function code and other parameters. > 3. Hypervisor switches context to native app vCPU > 4. When native app finishes request handling it calls special syscall > app_exit() > 5. Hypervisor analyses return code, updates requester vCPU state (if needed), > switches back to that vCPU, unpauses it. > > Most of that was done at [2]. Most interesting part is in arch/arm/domain.c > There are functions call_el0_app() and return_from_el0_app() that do most > of the work. Also I have added syscalls handlers (in the same way, > as hypercalls are handled). You can find them at xen/arch/arm/syscall.c This workflow is actually kind of OK. I would not use the term "vcpu" for anything related to an el0 app. Maybe we need to introduce a new concept, such as "app_context" for example. But I would not want to confuse "vcpu" which is the runtime environment exposed to guests, with the el0 Xen context. A vcpu is expected to be running simultenously with other vcpus of the same domain or different domains. The scheduler is expected to choose when it is supposed to be running. On the other end, an el0 app runs to handle/emulate a single request from a guest vcpu, which will be paused until the el0 app finishes. After that, the guest vcpu will resume. > At this moment entry point is hardcoded and you need to update it every time > you rebuild native application. Also there are no actual parameters passed. > Also, whole code is a piece of gosa, because it was first time I hacked XEN. :-) I would start by introducing a proper way to pass parameters and return values. > I don't want to repeat benchmark results, because they already was posted in > ML. > You can find them at [3]. > > I understand that I have missed many things: > > 1. How to ship and load native app, because some of them will be needed even > before dom0 is created. I envision something like Linux's insmod, but I suggest postponing this problem. At the moment, it would be fine to assume that all apps need to be built statically and cannot be loaded at runtime. > 2. How to distinguish multiple native apps Each apps need to specify a range of MMIO/SMC handlers. Xen will invoke the right one. > 3. Concurrency in native apps This is an interesting problem: what do we do if two guest vcpus make simultenous requests that need to be handled by the same app? Technically, we could run the same app twice on two different pcpus simultenously. But then, the apps would need to be able to cope with concurrency (spin_locks, etc.) From Xen point of view, it should be OK though. > 4. How to restart misbehaved apps. A related question is the following: do we expect to allocate each app once at boot or once per guest? Apps need to have some per-domain context, but it could be passed from Xen to the app on a shared page, possibly reducing the need for allocating the same app once per guest? > But at this moment I want to discuss basic approach. If there are will be no > objections against basic concept, then we can develop details. > > [1] https://github.com/lorc/xen_app_stub - native app > [2] https://github.com/lorc/xen/tree/el0_app - my branch with PoC > [3] http://marc.info/?l=xen-devel&m=149088856116797&w=2 - benchmark results _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel