Jailhouse and optee interaction
HI all, We use jailhouse hypervisor on our platforms. Standard configuration is root cell - linux, secondary cell - ThreadX. But now we want to add optee os support. How jailhouse can interact with optee os. What is the common way to run optee os in jailhouse. Thanks, Vladimir -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it. -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/CAKM86uE4n7wWu2UZYr%2BhGOBrZ4d5eHsJhOND7pmd4apNZzhLBA%40mail.gmail.com. smime.p7s Description: S/MIME Cryptographic Signature
RE: JAILHOUSE_HC_CELL_CREATE stuck
My goal is to launch root cell from RTOS. And in inmate cell must run linux. I use jailhouse driver as reference code. Now I works not on real board (with RTOS) but on another ARMV8 based board with Linux. As I understood PSCI in configured in Linux on this board. Thanks, Vladimir -Original Message- From: Jan Kiszka [mailto:jan.kis...@siemens.com] Sent: Tuesday, November 6, 2018 4:58 PM To: Vladimir Neyelov ; jailhouse-dev@googlegroups.com Subject: Re: JAILHOUSE_HC_CELL_CREATE stuck On 06.11.18 15:36, Vladimir Neyelov wrote: > Hi Jan, > > Simply I use same function that you use in jailhouse driver - cpu_down > for inmate CPU before hypercall JAILHOUSE_HC_CELL_CREATE. Hmm, are you reimplementing the driver module? Anyway, my comment was rather targeting at the effect of that function: Normally, the kernel would put the CPU into power-down state after offlining it, e.g. via PSCI. PSCI is properly intercepted by Jailhouse and this command will not execute physically once Jailhouse is active. If offlining should work differently on your board (we had a couple of cases on legacy ARMv7), this assumption may break. Jan > > Thanks, > Vladimir > > -Original Message- > From: Jan Kiszka [mailto:jan.kis...@siemens.com] > Sent: Tuesday, November 6, 2018 3:34 PM > To: Vladimir Neyelov ; > jailhouse-dev@googlegroups.com > Subject: Re: JAILHOUSE_HC_CELL_CREATE stuck > > On 06.11.18 13:29, 'Vladimir Neyelov' via Jailhouse wrote: >> Hello all, >> >> I am trying to porting jailhouse on our arm64 base board. I created >> configuration for inmate cell and >> >> trying to launch it. I have cpu bit mask 0xf for root cell and 0x8 >> for inmate. >> >> This is my log: >> >> >> >> NONROOT: handle_hvc >> >> NONROOT: bitmap f, max_cpu_id 63 >> >> NONROOT: suspend CPU 1 >> >> NONROOT: cpu 1 try suspend >> >> SGI_EVENT >> >> NONROOT: cpu 1 suspended >> >> NONROOT: suspend CPU 2 >> >> NONROOT: cpu 2 try suspend >> >> SGI_EVENT >> >> NONROOT: cpu 2 suspended >> >> NONROOT: suspend CPU 3 >> >> NONROOT: cpu 3 try suspend >> >> >> >> In function cell_create called cell_suspend(_cell) and stuck. >> >> As you can see code performed on CPU 0 and it try suspend CPU 1,2 and >> 3. 1 and 2 suspended successfully. >> >> But in case of CPU 3 it stuck in function arch_suspend_cpu on line: >> >> while (!target_data->cpu_suspended) >> >> cpu_relax(); >> >> >> >> SGI_EVENT sent to CPU 3 but doesn’t reach to exception routine. >> >> CPU 3 is in down state in linux. Why we need suspend CPU 3 that now >> belong to inmate cell ? >> > > Could it be that this "down state" was actually physically executed? > Then the hypervisor lost control over the CPU, and it would explain > why the SGI does not make. > > As we do not know the board nor the patches to wrote, it's hard to be > more concrete. > > Jan > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate > Competence Center Embedded Linux > -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
RE: JAILHOUSE_HC_CELL_CREATE stuck
Hi Jan, Jailhouse driver removes inmate cell CPU from root cell , shutdown inmate cell CPU and call JAILHOUSE_HC_CELL_CREATE. But driver removes cell CPU locally not in hypervisor data structure. Now inmate CPU is in offline and cannot receive SGI events. Thanks, Vladimir -Original Message- From: Vladimir Neyelov [mailto:vladimir.neye...@broadcom.com] Sent: Tuesday, November 6, 2018 4:36 PM To: 'Jan Kiszka' ; 'jailhouse-dev@googlegroups.com' Subject: RE: JAILHOUSE_HC_CELL_CREATE stuck Hi Jan, Simply I use same function that you use in jailhouse driver - cpu_down for inmate CPU before hypercall JAILHOUSE_HC_CELL_CREATE. Thanks, Vladimir -Original Message- From: Jan Kiszka [mailto:jan.kis...@siemens.com] Sent: Tuesday, November 6, 2018 3:34 PM To: Vladimir Neyelov ; jailhouse-dev@googlegroups.com Subject: Re: JAILHOUSE_HC_CELL_CREATE stuck On 06.11.18 13:29, 'Vladimir Neyelov' via Jailhouse wrote: > Hello all, > > I am trying to porting jailhouse on our arm64 base board. I created > configuration for inmate cell and > > trying to launch it. I have cpu bit mask 0xf for root cell and 0x8 for > inmate. > > This is my log: > > > > NONROOT: handle_hvc > > NONROOT: bitmap f, max_cpu_id 63 > > NONROOT: suspend CPU 1 > > NONROOT: cpu 1 try suspend > > SGI_EVENT > > NONROOT: cpu 1 suspended > > NONROOT: suspend CPU 2 > > NONROOT: cpu 2 try suspend > > SGI_EVENT > > NONROOT: cpu 2 suspended > > NONROOT: suspend CPU 3 > > NONROOT: cpu 3 try suspend > > > > In function cell_create called cell_suspend(_cell) and stuck. > > As you can see code performed on CPU 0 and it try suspend CPU 1,2 and > 3. 1 and 2 suspended successfully. > > But in case of CPU 3 it stuck in function arch_suspend_cpu on line: > > while (!target_data->cpu_suspended) > > cpu_relax(); > > > > SGI_EVENT sent to CPU 3 but doesn’t reach to exception routine. > > CPU 3 is in down state in linux. Why we need suspend CPU 3 that now > belong to inmate cell ? > Could it be that this "down state" was actually physically executed? Then the hypervisor lost control over the CPU, and it would explain why the SGI does not make. As we do not know the board nor the patches to wrote, it's hard to be more concrete. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
RE: JAILHOUSE_HC_CELL_CREATE stuck
Hi Jan, Simply I use same function that you use in jailhouse driver - cpu_down for inmate CPU before hypercall JAILHOUSE_HC_CELL_CREATE. Thanks, Vladimir -Original Message- From: Jan Kiszka [mailto:jan.kis...@siemens.com] Sent: Tuesday, November 6, 2018 3:34 PM To: Vladimir Neyelov ; jailhouse-dev@googlegroups.com Subject: Re: JAILHOUSE_HC_CELL_CREATE stuck On 06.11.18 13:29, 'Vladimir Neyelov' via Jailhouse wrote: > Hello all, > > I am trying to porting jailhouse on our arm64 base board. I created > configuration for inmate cell and > > trying to launch it. I have cpu bit mask 0xf for root cell and 0x8 for > inmate. > > This is my log: > > > > NONROOT: handle_hvc > > NONROOT: bitmap f, max_cpu_id 63 > > NONROOT: suspend CPU 1 > > NONROOT: cpu 1 try suspend > > SGI_EVENT > > NONROOT: cpu 1 suspended > > NONROOT: suspend CPU 2 > > NONROOT: cpu 2 try suspend > > SGI_EVENT > > NONROOT: cpu 2 suspended > > NONROOT: suspend CPU 3 > > NONROOT: cpu 3 try suspend > > > > In function cell_create called cell_suspend(_cell) and stuck. > > As you can see code performed on CPU 0 and it try suspend CPU 1,2 and > 3. 1 and 2 suspended successfully. > > But in case of CPU 3 it stuck in function arch_suspend_cpu on line: > > while (!target_data->cpu_suspended) > > cpu_relax(); > > > > SGI_EVENT sent to CPU 3 but doesn’t reach to exception routine. > > CPU 3 is in down state in linux. Why we need suspend CPU 3 that now > belong to inmate cell ? > Could it be that this "down state" was actually physically executed? Then the hypervisor lost control over the CPU, and it would explain why the SGI does not make. As we do not know the board nor the patches to wrote, it's hard to be more concrete. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
JAILHOUSE_HC_CELL_CREATE stuck
Hello all, I am trying to porting jailhouse on our arm64 base board. I created configuration for inmate cell and trying to launch it. I have cpu bit mask 0xf for root cell and 0x8 for inmate. This is my log: NONROOT: handle_hvc NONROOT: bitmap f, max_cpu_id 63 NONROOT: suspend CPU 1 NONROOT: cpu 1 try suspend SGI_EVENT NONROOT: cpu 1 suspended NONROOT: suspend CPU 2 NONROOT: cpu 2 try suspend SGI_EVENT NONROOT: cpu 2 suspended NONROOT: suspend CPU 3 NONROOT: cpu 3 try suspend In function cell_create called cell_suspend(_cell) and stuck. As you can see code performed on CPU 0 and it try suspend CPU 1,2 and 3. 1 and 2 suspended successfully. But in case of CPU 3 it stuck in function arch_suspend_cpu on line: while (!target_data->cpu_suspended) cpu_relax(); SGI_EVENT sent to CPU 3 but doesn’t reach to exception routine. CPU 3 is in down state in linux. Why we need suspend CPU 3 that now belong to inmate cell ? Thanks, Vladimir -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: jaihouse porting on arm64 board
Hi Jan,
Great advice it helped me. Now we have a working root cell on our board.
Next step inmate cell.
Thanks,
Vladimir
On Thursday, October 11, 2018, Jan Kiszka wrote:
> On 11.10.18 16:14, Vladimir Neyelov wrote:
>
>> Hi Jan,
>>
>> .mem_regions = {
>>> {
>>> .phys_start = 0x80008000,
>>> .virt_start = 0x80008000,
>>> .size = 0x34DD000,
>>> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
>>> JAILHOUSE_MEM_IO,
>>>
>>
>> This grants the root cell direct access to the GIC - not good...
>>>
>>
>> Do you mean - remove gic registers from memory regions ?
>>
>
> Yes. Those need to be handled by the hypervisor. That one is already
> ensuring the that GICC is mapped for the guest (to the GICV), but the GICD
> requires trapping. And GICH is surely no guest business.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux
>
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
RE: jaihouse porting on arm64 board
Hi Jan,
> .mem_regions = {
> {
> .phys_start = 0x80008000,
> .virt_start = 0x80008000,
> .size = 0x34DD000,
> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> JAILHOUSE_MEM_IO,
> This grants the root cell direct access to the GIC - not good...
Do you mean - remove gic registers from memory regions ?
Vladimir
-Original Message-
From: Jan Kiszka [mailto:jan.kis...@siemens.com]
Sent: Thursday, October 11, 2018 4:22 PM
To: Vladimir Neyelov ; Ralf Ramsauer
; jailhouse-dev@googlegroups.com
Subject: Re: jaihouse porting on arm64 board
On 11.10.18 14:59, 'Vladimir Neyelov' via Jailhouse wrote:
> This is my configuration file based on other examples of arm64 boards
> configurations.
> I tried to remove virtual PCI device but it doesn’t influence on this
> issue.
>
> #include
> #include
>
> #define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))
>
> struct {
> struct jailhouse_system header;
> __u64 cpus[1];
> struct jailhouse_memory mem_regions[4];
> struct jailhouse_irqchip irqchips[1];
> struct jailhouse_pci_device pci_devices[1]; } __attribute__((packed))
> config = {
> .header = {
> .signature = JAILHOUSE_SYSTEM_SIGNATURE,
> .revision = JAILHOUSE_CONFIG_REVISION,
> .flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE,
> .hypervisor_memory = {
> .phys_start = 0x1900,
> .size = 0x0040,
> },
> .debug_console = {
> .address = 0xff800640,
> .size = 0x40,
> .type = JAILHOUSE_CON_TYPE_63XX,
> .flags = JAILHOUSE_CON_ACCESS_MMIO |
>JAILHOUSE_CON_REGDIST_4,
> },
> .platform_info = {
> .pci_mmconfig_base = 0x8004,
> .pci_mmconfig_end_bus = 0,
> .pci_is_virtual = 1,
> .pci_domain = -1,
> .arm = {
> .gic_version = 2,
> .gicd_base = 0x81001000,
> .gicc_base = 0x81002000,
> .gich_base = 0x81004000,
> .gicv_base = 0x81006000,
> .maintenance_irq = 25,
> },
> },
> .root_cell = {
> .name = "bcell",
>
> .cpu_set_size = sizeof(config.cpus),
> .num_memory_regions = ARRAY_SIZE(config.mem_regions),
> .num_irqchips = ARRAY_SIZE(config.irqchips),
> .num_pci_devices = ARRAY_SIZE(config.pci_devices),
>
> .vpci_irq_base = 128-32,
> },
> },
>
> .cpus = {
> 0xf,
> },
>
> .mem_regions = {
> {
> .phys_start = 0x80008000,
> .virt_start = 0x80008000,
> .size = 0x34DD000,
> .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
> JAILHOUSE_MEM_IO,
This grants the root cell direct access to the GIC - not good...
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence
Center Embedded Linux
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jailhouse-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
RE: jaihouse porting on arm64 board
This is my configuration file based on other examples of arm64 boards
configurations.
I tried to remove virtual PCI device but it doesn’t influence on this issue.
#include
#include
#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))
struct {
struct jailhouse_system header;
__u64 cpus[1];
struct jailhouse_memory mem_regions[4];
struct jailhouse_irqchip irqchips[1];
struct jailhouse_pci_device pci_devices[1];
} __attribute__((packed)) config = {
.header = {
.signature = JAILHOUSE_SYSTEM_SIGNATURE,
.revision = JAILHOUSE_CONFIG_REVISION,
.flags = JAILHOUSE_SYS_VIRTUAL_DEBUG_CONSOLE,
.hypervisor_memory = {
.phys_start = 0x1900,
.size = 0x0040,
},
.debug_console = {
.address = 0xff800640,
.size = 0x40,
.type = JAILHOUSE_CON_TYPE_63XX,
.flags = JAILHOUSE_CON_ACCESS_MMIO |
JAILHOUSE_CON_REGDIST_4,
},
.platform_info = {
.pci_mmconfig_base = 0x8004,
.pci_mmconfig_end_bus = 0,
.pci_is_virtual = 1,
.pci_domain = -1,
.arm = {
.gic_version = 2,
.gicd_base = 0x81001000,
.gicc_base = 0x81002000,
.gich_base = 0x81004000,
.gicv_base = 0x81006000,
.maintenance_irq = 25,
},
},
.root_cell = {
.name = "bcell",
.cpu_set_size = sizeof(config.cpus),
.num_memory_regions = ARRAY_SIZE(config.mem_regions),
.num_irqchips = ARRAY_SIZE(config.irqchips),
.num_pci_devices = ARRAY_SIZE(config.pci_devices),
.vpci_irq_base = 128-32,
},
},
.cpus = {
0xf,
},
.mem_regions = {
{
.phys_start = 0x80008000,
.virt_start = 0x80008000,
.size = 0x34DD000,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_IO,
},
{
.phys_start = 0xff80,
.virt_start = 0xff80,
.size = 0x80,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_IO,
},
/* RAM */ {
.phys_start = 0x0,
.virt_start = 0x0,
.size = 0x2000,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE |
JAILHOUSE_MEM_EXECUTE,
},
/* IVSHMEM shared memory region for 00:00.0 */ {
.phys_start = 0x3fb0,
.virt_start = 0x3fb0,
.size = 0x10,
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE,
},
},
.irqchips = {
/* GIC */ {
.address = 0x81001000,
.pin_base = 32,
.pin_bitmap = {
0x, 0x, 0x, 0x,
},
},
},
.pci_devices = {
/* 0001:00:00.0 */ {
.type = JAILHOUSE_PCI_TYPE_IVSHMEM,
.domain = 1,
.bdf = 0 << 3,
.bar_mask = {
0xff00, 0x, 0x,
0x, 0x, 0x,
},
.shmem_region = 3,
.shmem_protocol = JAILHOUSE_SHMEM_PROTO_VETH,
},
},
};
-Original Message-
From: Ralf Ramsauer [mailto:ralf.ramsa...@oth-regensburg.de]
Sent: Thursday, October 11, 2018 3:45 PM
To: Vladimir Neyelov ;
jailhouse-dev@googlegroups.com
Subject: Re: jaihouse porting on arm64 board
On 10/11/18 2:30 PM, 'Vladimir Neyelov' via Jailhouse wrote:
> Hello all,
>
> I am trying to porting jailhouse on our arm64 base board. I created
> configuration for root cell and
>
> trying to launch root cell.
>
>
>
> This is output form uart:
>
>
>
> Initializing Jailhouse hypervisor v0.9.1 (134-g006918ca-dirty) on CPU
> 3
>
>
jaihouse porting on arm64 board
Hello all, I am trying to porting jailhouse on our arm64 base board. I created configuration for root cell and trying to launch root cell. This is output form uart: Initializing Jailhouse hypervisor v0.9.1 (134-g006918ca-dirty) on CPU 3 Code location: 0xc0200800 Page pool usage after early setup: mem 39/996, remap 0/131072 Initializing processors: CPU 3... OK CPU 0... OK CPU 2... OK CPU 1... OK Initializing unit: irqchip Initializing unit: PCI Adding virtual PCI device 00:00.0 to cell "bcell" Page pool usage after late setup: mem 58/996, remap 5/131072 JAIL: exit loop JAIL: exit loop JAIL: exit loop Activating hypervisor JAIL: exit loop JAIL: exit loop -> is my print in jailhouse driver in function enter_hypervisor after exiting from entry to hypervisor. But after it linux stuck. Serial input/output doesn’t work. Debugger shows that linux run in el1_irg and received non stop spurious interrupt 1023. Why it happen. As I understand all interrupt firstly received by jailhouse and after forwarded to linux(EL1). Spurious interrupt must not forwarded to linux. Thanks, Vladimir -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
