Re: Assertion failure error in kernel vgic interrupt processing
On Thu May 03, 2018 at 11:18:03 +0800, nico.hacker wrote: > The previous test is not enough to cause error, this error is not related to > peripherals, I have removed almost all peripheral driversIn the > previous test, I added a delay operation before the vm2 start, so that vm1 > completely up and then start vm2, this will greatly reduce the probability of > the error. Now I removed this delay, vm1 and vm2 started simultaneously, then > the error will occur frequently (about 1/10 probability). The error is > generated when kernel processes vtimer interrupts. Under normal > circumstances, after processing a vtimer interrupt, the vcpu thread will > return to uvmm and then execute prepare_guest_entry(). This operation will > eventually enter the kernel through the system call to set the state of vcpu > thread, but when the error occurs, I find that the vcpu thread does not > execute vcpu_resume, it processes two consecutive vtimer interrupts. As a > result, the assertion fails. Ok, I see. I've never experienced this, however I will try running VMs aggressivly concurrent and see what happens. > Is the kernel able to guarantee that the complete process of handling vgic > and vtimer interrupts (including the process in uvmm) is not interrupted by a > new interrupt? Yes, this is supposed to be the case. > Here's my configuration: You're aware that you seem to give both VMs access to the same UART device? (Will probably have nothing to do with the assertion.) Adam ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Re: Assertion failure error in kernel vgic interrupt processing
Hi,The previous test is not enough to cause error, this error is not related to peripherals, I have removed almost all peripheral driversIn the previous test, I added a delay operation before the vm2 start, so that vm1 completely up and then start vm2, this will greatly reduce the probability of the error. Now I removed this delay, vm1 and vm2 started simultaneously, then the error will occur frequently (about 1/10 probability). The error is generated when kernel processes vtimer interrupts. Under normal circumstances, after processing a vtimer interrupt, the vcpu thread will return to uvmm and then execute prepare_guest_entry(). This operation will eventually enter the kernel through the system call to set the state of vcpu thread, but when the error occurs, I find that the vcpu thread does not execute vcpu_resume, it processes two consecutive vtimer interrupts. As a result, the assertion fails. Is the kernel able to guarantee that the complete process of handling vgic and vtimer interrupts (including the process in uvmm) is not interrupted by a new interrupt? Here's my configuration:-uvmm.nedvmm.start_vm{ id=1, mem=256, vbus=io_busses.vm_hw, bootargs=common_bootargs .. " ramdisk_size=9100 root=/dev/ram", kernel="rom/Image.gz", rd="rom/ramdisk-arm.rd", fdt="rom/virt-board.dtb", prio=nil, cpus=0x1};vmm.start_vm{ id=2, mem=256, vbus=io_busses.vm_nohw, bootargs=common_bootargs .. " ramdisk_size=9100 root=/dev/ram", kernel="rom/Image.gz", rd="rom/ramdisk-arm.rd", fdt="rom/virt-board.dtb", prio=nil, cpus=0x1};--io.cfg-- vi:ft=lualocal Res = Io.Reslocal Hw = Io.Hwlocal add_children = Io.Dt.add_childrenadd_children(Io.system_bus(), function() VGIC = Hw.Device(function() Property.hid = "arm-gicc"; Resource.reg0 = Res.mmio(0x12006000, 0x12007fff); Property.flags = Io.Hw_device_DF_multi_vbus; end) VM1_DEV = Hw.Device(function() Property.hid = "vm1_dev"; Resource.reg0 = Res.mmio(0x7010, 0x70100FFF); --UART end) VM2_DEV = Hw.Device(function() Property.hid = "vm2_dev"; Resource.reg0 = Res.mmio(0x7010, 0x70100FFF); --UART end)end)vm_hw.vbusIo.add_vbusses{ vm_hw = Io.Vi.System_bus(function() VGIC = wrap(Io.system_bus():match("arm-gicc")); VM1_DEV = wrap(Io.system_bus():match("vm1_dev")); end);}vm_nohw.vbusIo.add_vbusses{ vm_nohw = Io.Vi.System_bus(function() VGIC = wrap(Io.system_bus():match("arm-gicc")); VM2_DEV = wrap(Io.system_bus():match("vm2_dev")); end);}Nico于 2018-04-11 05:59:58,Adam Lackorzynski写道:On Mon Apr 09, 2018 at 22:26:38 +0800, nico wrote: > Sorry, the last email did not specify this information. > I gave VM1 all the hardware resources, and it used eight cores. VM2 only has > uart resources (also timer, gic resources), using a single core. I tried to > give both VM1 and VM2 only the necessary hardware resources( uart, timer, gic), > and that error won't happen. Ok, that's interesting. So that means that some of the passed through hardware is causing this. Would you be able to find out which device is causing this, by selectively enabling/disabling devices? Adam > On 4/9/2018 06:28ï¼ Adam Lackorzynski wroteï¼ > > Hi Nico, > > On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote: > > Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is > running 64bit Linux. It is recommended to start two or three vms, that > will be easier to reproduce this problem. > > > So, I tried several things but cannot reproduce this issue. I assume > this are single-core VMs only? Any hardware passed through? > Just trying to narrow it down... ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers http://r.g.tom.com/kwap/r/app/other/suixinyou.png);background-repeat:no-repeat;background-position:left center;font-size:14px;background-size: 20px;height: 39px;line-height: 39px;padding-left: 25px;display:block;color:#33;text-decoration: none;" href="http://mail.tom.com/webmail-static/welcomesxy.html"; onmouseover="this.style.cssText='background-image:url(http://r.g.tom.com/kwap/r/app/other/suixinyou.png);background-repeat:no-repeat;background-position:left center;font-size:14px;background-size: 20px;height: 39px;line-height: 39px;padding-left: 27px;display:block;color:#4c4c4c; text-decoration:underline;'" onmouseout="this.style.cssText='background-image:url(http://r.g.tom.com/kwap/r/app/other/suixinyou.png);background-repeat:no-repeat;background-position:left center;font-size:14px;bac
Re: Assertion failure error in kernel vgic interrupt processing
On Mon Apr 09, 2018 at 22:26:38 +0800, nico wrote: > Sorry, the last email did not specify this information. > I gave VM1 all the hardware resources, and it used eight cores. VM2 only has > uart resources (also timer, gic resources), using a single core. I tried to > give both VM1 and VM2 only the necessary hardware resources( uart, timer, > gic), > and that error won't happen. Ok, that's interesting. So that means that some of the passed through hardware is causing this. Would you be able to find out which device is causing this, by selectively enabling/disabling devices? Adam > On 4/9/2018 06:28ï¼ Adam Lackorzynski wroteï¼ > > Hi Nico, > > On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote: > > Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is > running 64bit Linux. It is recommended to start two or three vms, that > will be easier to reproduce this problem. > > > So, I tried several things but cannot reproduce this issue. I assume > this are single-core VMs only? Any hardware passed through? > Just trying to narrow it down... ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Re: Assertion failure error in kernel vgic interrupt processing
Sorry, the last email did not specify this information. I gave VM1 all the hardware resources, and it used eight cores. VM2 only has uart resources (also timer, gic resources), using a single core. I tried to give both VM1 and VM2 only the necessary hardware resources( uart, timer, gic), and that error won't happen.Nico On 4/9/2018 06:28,Adam Lackorzynski wrote: Hi Nico,On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote: Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is running 64bit Linux. It is recommended to start two or three vms, that will be easier to reproduce this problem.So, I tried several things but cannot reproduce this issue. I assumethis are single-core VMs only? Any hardware passed through?Just trying to narrow it down... ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Re: Assertion failure error in kernel vgic interrupt processing
Hi Nico, On Fri Mar 30, 2018 at 10:15:33 +0800, nico wrote: > Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is > running 64bit Linux. It is recommended to start two or three vms, that > will be easier to reproduce this problem. So, I tried several things but cannot reproduce this issue. I assume this are single-core VMs only? Any hardware passed through? Just trying to narrow it down... Adam ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Re: Assertion failure error in kernel vgic interrupt processing
Hi Adam, I'm using an eight-core Cortex-A53 processor. The vm is running 64bit Linux. It is recommended to start two or three vms, that will be easier to reproduce this problem. Nico 在2018年03月30日 00:24,Adam Lackorzynski 写道: Hi, On Thu Mar 29, 2018 at 20:48:34 +0800, nico wrote: > Hi l4 hackers, > > I recently encountered an assertion failure error during the vm linux boot > process, the kernel error is at src/kern/arm/thread-arm-hyp.cpp. > -- > vcpu_vgic_upcall(unsigned virq) > { > .. > assert (state() & Thread_vcpu_user); > .. > } > -- > The thread reports this error while processing the vgic interrupt. I'm not sure > where the thread state has been changed. The most suspicious is the function > vcpu_enter_kernel_mode(), but the function is called in several places. I have > updated to the latest version (l4re-snapshot-18.03) and the error still > appears. > There is also a phenomenon that if multiple VMs are started, the frequency of > this error will increase a lot. Interesting. Could you please be a little bit more verbose, e.g. what type of CPU/SoC you're using and whether you're on 32 or 64bit? Would I be able to reproduce this? Adam ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Re: Assertion failure error in kernel vgic interrupt processing
Hi, On Thu Mar 29, 2018 at 20:48:34 +0800, nico wrote: > Hi l4 hackers, > > I recently encountered an assertion failure error during the vm linux boot > process, the kernel error is at src/kern/arm/thread-arm-hyp.cpp. > -- > vcpu_vgic_upcall(unsigned virq) > { >.. >assert (state() & Thread_vcpu_user); >.. > } > -- > The thread reports this error while processing the vgic interrupt. I'm not > sure > where the thread state has been changed. The most suspicious is the function > vcpu_enter_kernel_mode(), but the function is called in several places. I have > updated to the latest version (l4re-snapshot-18.03) and the error still > appears. > There is also a phenomenon that if multiple VMs are started, the frequency of > this error will increase a lot. Interesting. Could you please be a little bit more verbose, e.g. what type of CPU/SoC you're using and whether you're on 32 or 64bit? Would I be able to reproduce this? Adam ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
Assertion failure error in kernel vgic interrupt processing
Hi l4 hackers,I recently encountered an assertion failure error during the vm linux boot process, the kernel error is at src/kern/arm/thread-arm-hyp.cpp. -- vcpu_vgic_upcall(unsigned virq) { .. assert (state() & Thread_vcpu_user); .. }--The thread reports this error while processing the vgic interrupt. I'm not sure where the thread state has been changed. The most suspicious is the function vcpu_enter_kernel_mode(), but the function is called in several places. I have updated to the latest version (l4re-snapshot-18.03) and the error still appears.There is also a phenomenon that if multiple VMs are started, the frequency of this error will increase a lot. Thanks in advance for any suggestions.RegardsNico ___ l4-hackers mailing list l4-hackers@os.inf.tu-dresden.de http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers