Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-06 Thread Rishi
On Wed, Nov 7, 2018 at 12:16 AM Rishi <2rushike...@gmail.com> wrote: > > > On Tue, Nov 6, 2018 at 10:41 PM Rishi <2rushike...@gmail.com> wrote: > >> >> >> On Tue, Nov 6, 2018 at 5:47 PM Wei Liu wrote: >> >>> On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote: >>> > >>> > So after knowing the

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-06 Thread Rishi
On Tue, Nov 6, 2018 at 10:41 PM Rishi <2rushike...@gmail.com> wrote: > > > On Tue, Nov 6, 2018 at 5:47 PM Wei Liu wrote: > >> On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote: >> > >> > So after knowing the stack trace, it appears that the CPU was getting >> stuck >> > for

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-06 Thread Rishi
On Tue, Nov 6, 2018 at 5:47 PM Wei Liu wrote: > On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote: > > > > So after knowing the stack trace, it appears that the CPU was getting > stuck > > for xen_hypercall_xen_version > > That hypercall is used when a PV kernel (re-)enables interrupts. See

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-06 Thread Wei Liu
On Tue, Nov 06, 2018 at 03:31:31PM +0530, Rishi wrote: > > So after knowing the stack trace, it appears that the CPU was getting stuck > for xen_hypercall_xen_version That hypercall is used when a PV kernel (re-)enables interrupts. See xen_irq_enable. The purpose is to force the kernel to switch

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-06 Thread Rishi
On Mon, Nov 5, 2018 at 6:32 PM Rishi <2rushike...@gmail.com> wrote: > > > On Mon, Nov 5, 2018 at 6:29 PM Rishi <2rushike...@gmail.com> wrote: > >> Yes, I'm taking out patches from 4.4 and actually do have a working 4.9 >> kernel along with blktap. Tested networking and disk IO in it. >> >> There

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Rishi
On Mon, Nov 5, 2018 at 6:29 PM Rishi <2rushike...@gmail.com> wrote: > Yes, I'm taking out patches from 4.4 and actually do have a working 4.9 > kernel along with blktap. Tested networking and disk IO in it. > > There are roughly 415 patches to 4.4 out of which some ~210+ are already > applied in

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Rishi
Yes, I'm taking out patches from 4.4 and actually do have a working 4.9 kernel along with blktap. Tested networking and disk IO in it. There are roughly 415 patches to 4.4 out of which some ~210+ are already applied in 4.9 and ~220+ are already applied in 4.14. I dont have numbers for 4.19 yet.

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Wei Liu
On Mon, Nov 05, 2018 at 05:18:43PM +0530, Rishi wrote: > Yes, I'm running it in a HVM domU for development purpose. What is your exact setup? Wei. > > On Mon, Nov 5, 2018 at 5:11 PM Wei Liu wrote: > > > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote: > > > Alright, I got the serial

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Rishi
Yes, I'm running it in a HVM domU for development purpose. On Mon, Nov 5, 2018 at 5:11 PM Wei Liu wrote: > On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote: > > Alright, I got the serial console and following is the crash log. Thank > you > > for pointing that out. > > > > [ 133.594852]

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Wei Liu
On Mon, Nov 05, 2018 at 04:58:35PM +0530, Rishi wrote: > Alright, I got the serial console and following is the crash log. Thank you > for pointing that out. > > [ 133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! > [ksoftirqd/2:22] > > [ 133.599232] Kernel panic - not syncing:

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Rishi
Alright, I got the serial console and following is the crash log. Thank you for pointing that out. [ 133.594852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:22] [ 133.599232] Kernel panic - not syncing: softlockup: hung tasks [ 133.602275] CPU: 2 PID: 22 Comm: ksoftirqd/2

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Wei Liu
On Mon, Nov 05, 2018 at 04:26:03PM +0530, Rishi wrote: > Nothing is getting logged. I suspect its because the kernel just goes in > halt state. > > /var/log/kern.log /var/log/messages /var/log/xen/hypervisor.log > > All of these files have nothing appended. > > I can fetch info from kernel

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Rishi
Nothing is getting logged. I suspect its because the kernel just goes in halt state. /var/log/kern.log /var/log/messages /var/log/xen/hypervisor.log All of these files have nothing appended. I can fetch info from kernel debugfs, if you point me to required area, The dom0 kernel works fine

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-05 Thread Wei Liu
On Sun, Nov 04, 2018 at 06:37:36PM +0530, Rishi wrote: > I've built a dom0 kernel 4.14 with SMP support. The dom0 kernel crashes > when I'm downloading a large file on host. It does not crash if I have > nosmp boot option on xen command line. > > my .config SMP options are > > [root@f6029920339a

Re: [Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-04 Thread Rishi
Found the references of these in https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg03120.html proposal to split PV, PVHVM and PVH code in kernel. There is no mention of any change or requirement from Xen perspective. Any other way to track this problem? On Sun, Nov 4, 2018 at 6:37

[Xen-devel] Dom0 kernel 4.14 with SMP randomly crashing

2018-11-04 Thread Rishi
I've built a dom0 kernel 4.14 with SMP support. The dom0 kernel crashes when I'm downloading a large file on host. It does not crash if I have nosmp boot option on xen command line. my .config SMP options are [root@f6029920339a wip-kernel-4.14.78]# grep SMP .config CONFIG_X86_64_*SMP*=y