Re: Assertion failure error in kernel vgic interrupt processing

[Adam Lackorzynski]

On Thu May 03, 2018 at 11:18:03 +0800, nico.hacker wrote:
> The previous test is not enough to cause error, this error is not related to 
> peripherals, I have removed almost all peripheral driversIn the 
> previous test, I added a delay operation before the vm2 start, so that vm1 
> completely up and then start vm2, this will greatly reduce the probability of 
> the error. Now I removed this delay, vm1 and vm2 started simultaneously, then 
> the error will occur frequently (about 1/10 probability). The error is 
> generated when kernel processes vtimer interrupts. Under normal 
> circumstances, after processing a vtimer interrupt, the vcpu thread will 
> return to uvmm and then execute prepare_guest_entry(). This operation will 
> eventually enter the kernel through the system call to set the state of vcpu 
> thread, but when the error occurs, I find that the vcpu thread does not 
> execute vcpu_resume, it processes two consecutive vtimer interrupts. As a 
> result, the assertion fails.

Ok, I see. I've never experienced this, however I will try running VMs
aggressivly concurrent and see what happens.

> Is the kernel able to guarantee that the complete process of handling vgic 
> and vtimer interrupts (including the process in uvmm) is not interrupted by a 
> new interrupt?

Yes, this is supposed to be the case.

> Here's my configuration:

You're aware that you seem to give both VMs access to the same UART
device? (Will probably have nothing to do with the assertion.)




Adam

___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

Re: Assertion failure error in kernel vgic interrupt processing

[nico . hacker]

Hi,The previous test is not enough to cause error, this 
error is not related to peripherals, I have removedalmost all peripheral 
driversIn the previous test, I added a delay operation before the 
vm2 start, so that vm1 completely up and then start vm2, this will greatly 
reduce the probability of the error. Now I removed this delay, vm1 and vm2 
started simultaneously, then the error will occur frequently (about 1/10 
probability). The error is generated when kernel processes vtimer interrupts. 
Under normal circumstances, after processing a vtimer interrupt, the vcpu 
thread will return to uvmm and then execute prepare_guest_entry(). This 
operation will eventually enter the kernel through the system call to set the 
state of vcpu thread, but when the error occurs, I find that the vcpu thread 
does not execute vcpu_resume, it processes two consecutive vtimer interrupts. 
As a result, the assertion fails.Is the kernel able to 
guarantee that the complete process of handling vgic and vtimer interrupts 
(including the process in uvmm) is not interrupted by a new interrupt?Here's my configuration:-uvmm.nedvmm.start_vm{ id=1, mem=256, vbus=io_busses.vm_hw, bootargs=common_bootargs ..  
ramdisk_size=9100 root=/dev/ram, 
kernel=rom/Image.gz, 
rd=rom/ramdisk-arm.rd, 
fdt=rom/virt-board.dtb, prio=nil, cpus=0x1};vmm.start_vm{ id=2, mem=256, 
vbus=io_busses.vm_nohw, bootargs=common_bootargs ..  
ramdisk_size=9100 root=/dev/ram, 
kernel=rom/Image.gz, 
rd=rom/ramdisk-arm.rd, 
fdt=rom/virt-board.dtb, prio=nil, cpus=0x1};--io.cfg-- vi:ft=lualocal Res = Io.Reslocal Hw = 
Io.Hwlocal add_children = Io.Dt.add_childrenadd_children(Io.system_bus(), function() VGIC = 
Hw.Device(function()  Property.hid = arm-gicc;  Resource.reg0 = Res.mmio(0x12006000, 0x12007fff); 
 Property.flags = Io.Hw_device_DF_multi_vbus; end) VM1_DEV = Hw.Device(function()  Property.hid = 
vm1_dev;  Resource.reg0 = Res.mmio(0x7010, 
0x70100FFF); --UART end) VM2_DEV = 
Hw.Device(function()  Property.hid = vm2_dev;  Resource.reg0 = Res.mmio(0x7010, 0x70100FFF); --UART end)end)vm_hw.vbusIo.add_vbusses{ vm_hw = 
Io.Vi.System_bus(function()  VGIC = 
wrap(Io.system_bus():match(arm-gicc));  VM1_DEV = 
wrap(Io.system_bus():match(vm1_dev)); end);}vm_nohw.vbusIo.add_vbusses{ vm_nohw = 
Io.Vi.System_bus(function()  VGIC = 
wrap(Io.system_bus():match(arm-gicc));  VM2_DEV = 
wrap(Io.system_bus():match(vm2_dev)); end);}Nico于  2018-04-11 
05:59:58,Adam Lackorzynski写道:On Mon Apr 09, 2018 at 22:26:38 +0800, 
nico wrote:
 Sorry, the last email did not specify this information. 
 I gave VM1 all the hardware resources, and it used eight cores. VM2 only 
has
 uart resources (also timer, gic resources), using a single core. I tried to
 give both VM1 and VM2 only the necessary hardware resources( uart, timer, 
gic),
 and that error won't happen.

Ok, that's interesting. So that means that some of the passed through
hardware is causing this. Would you be able to find out which device is
causing this, by selectively enabling/disabling devices?


Adam

 On 4/9/2018 06:28ï¼ Adam Lackorzynski wroteï¼ 
 
 Hi Nico,
 
 On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote:
 
 Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is
 running 64bit Linux. It is recommended to start two or three vms, 
that
 will be easier to reproduce this problem.
 
 
 So, I tried several things but cannot reproduce this issue. I assume
 this are single-core VMs only? Any hardware passed through?
 Just trying to narrow it down...

___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
http://r.g.tom.com/kwap/r/app/other/suixinyou.png);background-repeat:no-repeat;background-position:left
 center;font-size:14px;background-size: 20px;height: 39px;line-height: 
39px;padding-left: 25px;display:block;color:#33;text-decoration: none;" 
href="http://mail.tom.com/webmail-static/welcomesxy.html;  
onmouseover="this.style.cssText='background-image:url(http://r.g.tom.com/kwap/r/app/other/suixinyou.png);background-repeat:no-repeat;background-position:left
 center;font-size:14px;background-size: 20px;height: 39px;line-height: 
39px;padding-left: 27px;display:block;color:#4c4c4c; 
text-decoration:underline;'" 
onmouseout="this.style.cssText='background-image:url(http://r.g.tom.com/kwap/r/app/other/suixinyou.png);background-repeat:no-repeat;background-position:left
 center;font-size:14px;background-size: 20px;height: 39px;line-height: 
39px;padding-left: 

Re: Assertion failure error in kernel vgic interrupt processing

[Adam Lackorzynski]

On Mon Apr 09, 2018 at 22:26:38 +0800, nico wrote:
> Sorry, the last email did not specify this information. 
> I gave VM1 all the hardware resources, and it used eight cores. VM2 only has
> uart resources (also timer, gic resources), using a single core. I tried to
> give both VM1 and VM2 only the necessary hardware resources( uart, timer, 
> gic),
> and that error won't happen.

Ok, that's interesting. So that means that some of the passed through
hardware is causing this. Would you be able to find out which device is
causing this, by selectively enabling/disabling devices?


Adam

> On 4/9/2018 06:28ï¼ Adam Lackorzynski wroteï¼ 
> 
> Hi Nico,
> 
> On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote:
> 
> Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is
> running 64bit Linux. It is recommended to start two or three vms, that
> will be easier to reproduce this problem.
> 
> 
> So, I tried several things but cannot reproduce this issue. I assume
> this are single-core VMs only? Any hardware passed through?
> Just trying to narrow it down...

___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

Re: Assertion failure error in kernel vgic interrupt processing

[nico]

Sorry, the last email did not specify this information. I gave VM1 all the hardware resources, and it used eight cores. VM2 only has uart resources (also timer, gic resources), using a single core. I tried to give both VM1 and VM2 only the necessary hardware resources( uart, timer, gic), and that error won't happen.Nico


On 4/9/2018 06:28，Adam Lackorzynski wrote： 


Hi Nico,On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote: Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is running 64bit Linux. It is recommended to start two or three vms, that will be easier to reproduce this problem.So, I tried several things but cannot reproduce this issue. I assumethis are single-core VMs only? Any hardware passed through?Just trying to narrow it down...


___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

Re: Assertion failure error in kernel vgic interrupt processing

[Adam Lackorzynski]

Hi Nico,

On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote:
> Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is
> running 64bit Linux. It is recommended to start two or three vms, that
> will be easier to reproduce this problem.

So, I tried several things but cannot reproduce this issue. I assume
this are single-core VMs only? Any hardware passed through?
Just trying to narrow it down...


Adam

___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

Re: Assertion failure error in kernel vgic interrupt processing

[nico]

Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is running 64bit 
Linux. It is recommended to start two or three vms, that will be easier to 
reproduce this problem. Nico 在2018年03月30日 00:24，Adam Lackorzynski 写道： Hi, On 
Thu Mar 29, 2018 at 20:48:34 +0800, nico wrote: > Hi l4 hackers, > > I recently 
encountered an assertion failure error during the vm linux boot > process, the 
kernel error is at src/kern/arm/thread-arm-hyp.cpp. > 
-- >  
vcpu_vgic_upcall(unsigned virq) >  { >    .. >    assert (state() & 
Thread_vcpu_user); >    .. >  } > 
-- > The thread reports 
this error while processing the vgic interrupt. I'm not sure > where the thread 
state has been changed. The most suspicious is the function > 
vcpu_enter_kernel_mode(), but the function is called in several places. I have 
> updated to the latest version (l4re-snapshot-18.03) and the error still > 
appears. > There is also a phenomenon that if multiple VMs are started, the 
frequency of > this error will increase a lot. Interesting. Could you please be 
a little bit more verbose, e.g. what type of CPU/SoC you're using and whether 
you're on 32 or 64bit? Would I be able to reproduce this? Adam 
___ l4-hackers mailing list 
l4-hackers@os.inf.tu-dresden.de 
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers

Re: Assertion failure error in kernel vgic interrupt processing

[Adam Lackorzynski]

Hi,

On Thu Mar 29, 2018 at 20:48:34 +0800, nico wrote:
> Hi l4 hackers,
> 
> I recently encountered an assertion failure error during the vm linux boot
> process, the kernel error is at src/kern/arm/thread-arm-hyp.cpp. 
> --
>  vcpu_vgic_upcall(unsigned virq)
>  {
>..
>assert (state() & Thread_vcpu_user);
>..
>  }
> --
> The thread reports this error while processing the vgic interrupt. I'm not 
> sure
> where the thread state has been changed. The most suspicious is the function
> vcpu_enter_kernel_mode(), but the function is called in several places. I have
> updated to the latest version (l4re-snapshot-18.03) and the error still
> appears.
> There is also a phenomenon that if multiple VMs are started, the frequency of
> this error will increase a lot. 

Interesting. Could you please be a little bit more verbose, e.g. what
type of CPU/SoC you're using and whether you're on 32 or 64bit?
Would I be able to reproduce this?



Adam

___
l4-hackers mailing list
l4-hackers@os.inf.tu-dresden.de
http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers