RE: [EXT] Re: xen arm64 low power sleep support
On Wed, 13 Sep 2023, Anthony Chan wrote: > On Mon, 11 Sep 2023, Stefano Stabellini wrote: > > On Mon, 11 Sep 2023, Anthony Chan wrote: > > > On Wed, 6 Sep 2023, Stefano Stabellini wrote: > > > > On Wed, 6 Sep 2023, Anthony Chan wrote: > > > > > Thanks, I've tried patches that stemmed from that discussion but > > > > > unfortunately, doesn't resolve the issue. In fact, the > > > > > s2idle_loop branch might not be the problem at all. I > > > > > experimented with Xen to allow the 'idle-states' into the FDT and > > > > > prevented xen_guest_init on Linux from disabling the 'cpuidle' > > > > > driver (arch/arm/xen/enlighten.c). When I trigger a suspend, I > > > > > can see now another thread (believe it's the idle thread) call > > > > > into drivers/firmware/psci/psci.c:__psci_cpu_suspend and then the Xen > > > > > counterpart at xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend. > > > > > > > > OK but remember that Xen is not implementing do_psci_0_2_cpu_suspend > > > > correctly at the moment. Either we need to fix the Xen > > > > implementation, or we need to configure Linux so that it calls WFI > > > > instead > > > > of __psci_cpu_suspend. > > > > > > > > As a test, can you try to apply the attached patch to Xen as a > > > > tenative fix? Or you could change > > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend to call WFI instead > > > > of the PSCI operation (making sure to go to the entry_point instead of > > > > returning). > > > > > > Tried the patch and substituting a WFI for a PSCI op, but Xen still > > > watchdogs > > on the VMs in both cases. I noticed the other Linux generic arm 'cpu-idle' > > driver which used to do issue a WFI/cpu_do_idle isn't useable anymore > > either. > > I'm not sure if Xen may have used to rely on this generic driver to get the > > WFI. > > > > I was running out of ideas so I went back to look at the watchdog console > > log: > > > > (XEN) do_psci_0_2_cpu_suspend > > (XEN) Watchdog timer fired for domain 0 > > (XEN) Hardware Dom0 shutdown: watchdog rebooting machine > > > > Checking the code, it seems that the Xen watchdog is set by > > xen/common/sched/core.c:SCHEDOP_watchdog, which is called by > > tools/libs/ctrl/xc_domain.c:xc_watchdog. > > > > xc_watchdog is called by tools/misc/xenwatchdogd.c. Is it possible that this > > problem is entirely caused by the daemon xenwatchdogd running in the > > background? What happens if you kill xenwatchdogd and try again without it > > (even better not start it at all)? > Disabling that daemon resolved the watchdog timing out. Never noticed that > daemon running before. That clears a lot up and I think I understand what's > going on here now, thank you for the help. That's great! I am glad it got resolved.
RE: [EXT] Re: xen arm64 low power sleep support
On Mon, 11 Sep 2023, Stefano Stabellini wrote: > On Mon, 11 Sep 2023, Anthony Chan wrote: > > On Wed, 6 Sep 2023, Stefano Stabellini wrote: > > > On Wed, 6 Sep 2023, Anthony Chan wrote: > > > > Thanks, I've tried patches that stemmed from that discussion but > > > > unfortunately, doesn't resolve the issue. In fact, the > > > > s2idle_loop branch might not be the problem at all. I > > > > experimented with Xen to allow the 'idle-states' into the FDT and > > > > prevented xen_guest_init on Linux from disabling the 'cpuidle' > > > > driver (arch/arm/xen/enlighten.c). When I trigger a suspend, I > > > > can see now another thread (believe it's the idle thread) call > > > > into drivers/firmware/psci/psci.c:__psci_cpu_suspend and then the Xen > > > > counterpart at xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend. > > > > > > OK but remember that Xen is not implementing do_psci_0_2_cpu_suspend > > > correctly at the moment. Either we need to fix the Xen > > > implementation, or we need to configure Linux so that it calls WFI instead > > > of __psci_cpu_suspend. > > > > > > As a test, can you try to apply the attached patch to Xen as a > > > tenative fix? Or you could change > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend to call WFI instead > > > of the PSCI operation (making sure to go to the entry_point instead of > > > returning). > > > > Tried the patch and substituting a WFI for a PSCI op, but Xen still > > watchdogs > on the VMs in both cases. I noticed the other Linux generic arm 'cpu-idle' > driver which used to do issue a WFI/cpu_do_idle isn't useable anymore either. > I'm not sure if Xen may have used to rely on this generic driver to get the > WFI. > > I was running out of ideas so I went back to look at the watchdog console log: > > (XEN) do_psci_0_2_cpu_suspend > (XEN) Watchdog timer fired for domain 0 > (XEN) Hardware Dom0 shutdown: watchdog rebooting machine > > Checking the code, it seems that the Xen watchdog is set by > xen/common/sched/core.c:SCHEDOP_watchdog, which is called by > tools/libs/ctrl/xc_domain.c:xc_watchdog. > > xc_watchdog is called by tools/misc/xenwatchdogd.c. Is it possible that this > problem is entirely caused by the daemon xenwatchdogd running in the > background? What happens if you kill xenwatchdogd and try again without it > (even better not start it at all)? Disabling that daemon resolved the watchdog timing out. Never noticed that daemon running before. That clears a lot up and I think I understand what's going on here now, thank you for the help. CONFIDENTIALITY NOTICE: This e-mail, including any attachments, may contain information that is confidential and privileged. Any unauthorized disclosure, reproduction or use of this e-mail is prohibited. If you are not the intended recipient, please notify the sender by reply e-mail or telephone and permanently delete this e-mail and any reproductions immediately.
RE: [EXT] Re: xen arm64 low power sleep support
On Mon, 11 Sep 2023, Anthony Chan wrote: > On Wed, 6 Sep 2023, Stefano Stabellini wrote: > > On Wed, 6 Sep 2023, Anthony Chan wrote: > > > Thanks, I've tried patches that stemmed from that discussion but > > > unfortunately, doesn't resolve the issue. In fact, the s2idle_loop > > > branch might not be the problem at all. I experimented with Xen to allow > > > the > > > 'idle-states' into the FDT and prevented xen_guest_init on Linux from > > > disabling > > > the 'cpuidle' driver (arch/arm/xen/enlighten.c). When I trigger a > > > suspend, I > > > can see now another thread (believe it's the idle thread) call into > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend and then the Xen > > > counterpart at xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend. > > > > OK but remember that Xen is not implementing do_psci_0_2_cpu_suspend > > correctly at the moment. Either we need to fix the Xen implementation, or we > > need to configure Linux so that it calls WFI instead of __psci_cpu_suspend. > > > > As a test, can you try to apply the attached patch to Xen as a tenative > > fix? Or > > you could change drivers/firmware/psci/psci.c:__psci_cpu_suspend to call > > WFI instead of the PSCI operation (making sure to go to the entry_point > > instead of returning). > > Tried the patch and substituting a WFI for a PSCI op, but Xen still watchdogs > on the VMs in both cases. I noticed the other Linux generic arm 'cpu-idle' > driver which used to do issue a WFI/cpu_do_idle isn't useable anymore either. > I'm not sure if Xen may have used to rely on this generic driver to get the > WFI. I was running out of ideas so I went back to look at the watchdog console log: (XEN) do_psci_0_2_cpu_suspend (XEN) Watchdog timer fired for domain 0 (XEN) Hardware Dom0 shutdown: watchdog rebooting machine Checking the code, it seems that the Xen watchdog is set by xen/common/sched/core.c:SCHEDOP_watchdog, which is called by tools/libs/ctrl/xc_domain.c:xc_watchdog. xc_watchdog is called by tools/misc/xenwatchdogd.c. Is it possible that this problem is entirely caused by the daemon xenwatchdogd running in the background? What happens if you kill xenwatchdogd and try again without it (even better not start it at all)?
RE: [EXT] Re: xen arm64 low power sleep support
On Wed, 6 Sep 2023, Stefano Stabellini wrote: > On Wed, 6 Sep 2023, Anthony Chan wrote: > > Thanks, I've tried patches that stemmed from that discussion but > > unfortunately, doesn't resolve the issue. In fact, the s2idle_loop > > branch might not be the problem at all. I experimented with Xen to allow > > the > > 'idle-states' into the FDT and prevented xen_guest_init on Linux from > > disabling > > the 'cpuidle' driver (arch/arm/xen/enlighten.c). When I trigger a suspend, > > I > > can see now another thread (believe it's the idle thread) call into > > drivers/firmware/psci/psci.c:__psci_cpu_suspend and then the Xen > > counterpart at xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend. > > OK but remember that Xen is not implementing do_psci_0_2_cpu_suspend > correctly at the moment. Either we need to fix the Xen implementation, or we > need to configure Linux so that it calls WFI instead of __psci_cpu_suspend. > > As a test, can you try to apply the attached patch to Xen as a tenative fix? > Or > you could change drivers/firmware/psci/psci.c:__psci_cpu_suspend to call > WFI instead of the PSCI operation (making sure to go to the entry_point > instead of returning). Tried the patch and substituting a WFI for a PSCI op, but Xen still watchdogs on the VMs in both cases. I noticed the other Linux generic arm 'cpu-idle' driver which used to do issue a WFI/cpu_do_idle isn't useable anymore either. I'm not sure if Xen may have used to rely on this generic driver to get the WFI. CONFIDENTIALITY NOTICE: This e-mail, including any attachments, may contain information that is confidential and privileged. Any unauthorized disclosure, reproduction or use of this e-mail is prohibited. If you are not the intended recipient, please notify the sender by reply e-mail or telephone and permanently delete this e-mail and any reproductions immediately.
RE: [EXT] Re: xen arm64 low power sleep support
On Wed, 6 Sep 2023, Anthony Chan wrote:
> On Tue, 05 Sep 2023, Stefano Stabellini wrote:
> > On Thu, 31 Aug 2023, Anthony Chan wrote:
> > > On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> > > > On Wed, 30 Aug 2023, Anthony Chan wrote:
> > > > > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > > > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > My name is Tony and I've been researching/developing using
> > > > > > > Xen for potential upcoming uses in our embedded systems. I
> > > > > > > started with Xen using Xilinx tools about a year ago and
> > > > > > > still have lots to learn about what it can to do in the
> > > > > > > embedded space. So far, I've managed to integrate Xen and
> > > > > > > Linux into an existing product that exclusively runs
> > > > > > > bare-metal code on a ZynqMP SoC and migrate some of the
> > > > > > > functionality into custom Linux driver/userspace.
> > > > > > >
> > > > > > > I'm now looking at low power support, for now at least
> > > > > > > between Xen
> > > > > > > (4.16) and Linux (5.15) dom0. I've tried a few different
> > > > > > > Linux kernel configs around power management and each time I
> > > > > > > try to suspend from linux dom0 (via sysfs or systemctl), Xen
> > > > > > > will watchdog on dom0 guest.
> > > > > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what
> > > > > > > I can tell debugging through the linux suspend process is
> > > > > > > it's spinning in a 'suspend- to-idle' loop before it can get
> > > > > > > to issuing a 'WFI' or using PSCI interface to notify Xen.
> > > > > > > I'm beginning to suspect that 'low power' support for embedded
> > > > > > > arm64 just isn't quite there yet, or am I missing something
> > > > > > > in the configs?
> > > > > > >
> > > > > > > I realize this could very well be a Linux 'issue' but
> > > > > > > checking here first. I know Xen presents a flattened device
> > > > > > > tree to Linux without CPU idle-state nodes and maybe this is
> > > > > > > causing the linux guest to only do the suspend- to-idle
> > > > > > > mode? I should mention that I'm booting up using dom0less
> > > > > > > feature if that matters.
> > > > > >
> > > > > >
> > > > > > Hi Anthony,
> > > > > >
> > > > > > Assuming you are using the default Xen command line parameters
> > > > > > for Xilinx boards: sched=null vwfi=native, then if the guest
> > > > > > uses WFI, the CPU will execute WFI directly and go into low
> > > > > > power mode.
> > > > > Yes, using these command line params.
> > > > >
> > > > > > Given the issue you are describing, I am suspecting the guest
> > > > > > is not issuing
> > > > > > WFI: that is simple and known to work. Instead, I suspect that
> > > > > > Linux might be trying to use PSCI_suspend in a way that is not
> > > > > > supported or well- implemented by Xen.
> > > > > >
> > > > > > Can you check? You can add a printk in Linux
> > > > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > > > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > > > > Instrumented both places it doesn't appear to reach there. In
> > > > > kernel/power/suspend.c, there's a call to s2idle_loop that it's
> > > > > currently 'stuck' in and I think it doesn't get to the psci suspend
> > > > > your
> > > > > referring till afterwards, when suspend_ops->enter is called.
> > > > > Unfortunately, without any idle-states nodes in the FDT, the
> > > > > only suspend state Linux is defaults to is 'suspend to idle'.
> > > >
> > > > The fact that Linux uses "suspend to idle" is not a problem
> > > > because as I mentioned WFI or PSCI_suspent are not different on
> > > > Xen. That part is OK.
> > > What if using "suspend to idle" is preventing a WFI/PSCI_suspend?
> > > Which is what I believe I'm currently seeing in my setup. In
> > > kernel/power/suspend.c, suspend_devices_and_enter(), it gets into
> > > the this s2idle_loop and upon resuming from idle, it jumps past the
> > > point where I believe a WFI/PSCI_suspend can happen.
> > > if (state == PM_SUSPEND_TO_IDLE) {
> > > s2idle_loop();
> > > goto Platform_wake;
> > > }
> >
> > If that is the case, then it looks like a Linux bug. Maybe something
> > along these lines?
> >
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
> > .k%2F=05%7C01%7Canthonychan%40nureva.com%7Ccf5e956c6a4d42b1460a08
> > dbaf13e021%7C5aeb77fa643b473eaee0cb54a11ccba3%7C1%7C0%7C63829627212319
> > 7144%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT
> > iI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=lDfILPgJQQQPJOL44%2BcU
> > %2BN2RFOs8L6F4AE11kvOhOQE%3D=0
> > ernel.org%2Flinux-arm-
> > kernel%2F4665489.GXAFRqVoOG%40kreacher%2FT%2F%23m6edda92d0b5
> > dc09f8e05e7d6db3807501b7249f4=05%7C01%7Canthonychan%40n
> > ureva.com%7C144641906bef48b9180f08dbae7bc1d8%7C5aeb77fa643b47
> > 3eaee0cb54a11ccba3%7C1%7C0%7C638295618800415028%7CUnknown
> >
RE: [EXT] Re: xen arm64 low power sleep support
On Tue, 05 Sep 2023, Stefano Stabellini wrote:
> On Thu, 31 Aug 2023, Anthony Chan wrote:
> > On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> > > On Wed, 30 Aug 2023, Anthony Chan wrote:
> > > > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > My name is Tony and I've been researching/developing using
> > > > > > Xen for potential upcoming uses in our embedded systems. I
> > > > > > started with Xen using Xilinx tools about a year ago and
> > > > > > still have lots to learn about what it can to do in the
> > > > > > embedded space. So far, I've managed to integrate Xen and
> > > > > > Linux into an existing product that exclusively runs
> > > > > > bare-metal code on a ZynqMP SoC and migrate some of the
> > > > > > functionality into custom Linux driver/userspace.
> > > > > >
> > > > > > I'm now looking at low power support, for now at least
> > > > > > between Xen
> > > > > > (4.16) and Linux (5.15) dom0. I've tried a few different
> > > > > > Linux kernel configs around power management and each time I
> > > > > > try to suspend from linux dom0 (via sysfs or systemctl), Xen
> > > > > > will watchdog on dom0 guest.
> > > > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what
> > > > > > I can tell debugging through the linux suspend process is
> > > > > > it's spinning in a 'suspend- to-idle' loop before it can get
> > > > > > to issuing a 'WFI' or using PSCI interface to notify Xen.
> > > > > > I'm beginning to suspect that 'low power' support for embedded
> > > > > > arm64 just isn't quite there yet, or am I missing something
> > > > > > in the configs?
> > > > > >
> > > > > > I realize this could very well be a Linux 'issue' but
> > > > > > checking here first. I know Xen presents a flattened device
> > > > > > tree to Linux without CPU idle-state nodes and maybe this is
> > > > > > causing the linux guest to only do the suspend- to-idle
> > > > > > mode? I should mention that I'm booting up using dom0less
> > > > > > feature if that matters.
> > > > >
> > > > >
> > > > > Hi Anthony,
> > > > >
> > > > > Assuming you are using the default Xen command line parameters
> > > > > for Xilinx boards: sched=null vwfi=native, then if the guest
> > > > > uses WFI, the CPU will execute WFI directly and go into low
> > > > > power mode.
> > > > Yes, using these command line params.
> > > >
> > > > > Given the issue you are describing, I am suspecting the guest
> > > > > is not issuing
> > > > > WFI: that is simple and known to work. Instead, I suspect that
> > > > > Linux might be trying to use PSCI_suspend in a way that is not
> > > > > supported or well- implemented by Xen.
> > > > >
> > > > > Can you check? You can add a printk in Linux
> > > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > > > Instrumented both places it doesn't appear to reach there. In
> > > > kernel/power/suspend.c, there's a call to s2idle_loop that it's
> > > > currently 'stuck' in and I think it doesn't get to the psci suspend your
> > > > referring till afterwards, when suspend_ops->enter is called.
> > > > Unfortunately, without any idle-states nodes in the FDT, the
> > > > only suspend state Linux is defaults to is 'suspend to idle'.
> > >
> > > The fact that Linux uses "suspend to idle" is not a problem
> > > because as I mentioned WFI or PSCI_suspent are not different on
> > > Xen. That part is OK.
> > What if using "suspend to idle" is preventing a WFI/PSCI_suspend?
> > Which is what I believe I'm currently seeing in my setup. In
> > kernel/power/suspend.c, suspend_devices_and_enter(), it gets into
> > the this s2idle_loop and upon resuming from idle, it jumps past the
> > point where I believe a WFI/PSCI_suspend can happen.
> > if (state == PM_SUSPEND_TO_IDLE) {
> > s2idle_loop();
> > goto Platform_wake;
> > }
>
> If that is the case, then it looks like a Linux bug. Maybe something
> along these lines?
>
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
> .k%2F=05%7C01%7Canthonychan%40nureva.com%7Ccf5e956c6a4d42b1460a08
> dbaf13e021%7C5aeb77fa643b473eaee0cb54a11ccba3%7C1%7C0%7C63829627212319
> 7144%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT
> iI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=lDfILPgJQQQPJOL44%2BcU
> %2BN2RFOs8L6F4AE11kvOhOQE%3D=0
> ernel.org%2Flinux-arm-
> kernel%2F4665489.GXAFRqVoOG%40kreacher%2FT%2F%23m6edda92d0b5
> dc09f8e05e7d6db3807501b7249f4=05%7C01%7Canthonychan%40n
> ureva.com%7C144641906bef48b9180f08dbae7bc1d8%7C5aeb77fa643b47
> 3eaee0cb54a11ccba3%7C1%7C0%7C638295618800415028%7CUnknown
> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=%2Fv%2FHpI1ix3yhDJ
> 75oejWzDfUqB8SqOQzkU3clJzMOtw%3D=0
Thanks, I've tried patches that stemmed from that discussion but unfortunately,
doesn't resolve the issue. In
RE: [EXT] Re: xen arm64 low power sleep support
On Thu, 31 Aug 2023, Anthony Chan wrote:
> On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> > On Wed, 30 Aug 2023, Anthony Chan wrote:
> > > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > > Hi all,
> > > > >
> > > > > My name is Tony and I've been researching/developing using Xen
> > > > > for potential upcoming uses in our embedded systems. I started
> > > > > with Xen using Xilinx tools about a year ago and still have lots
> > > > > to learn about what it can to do in the embedded space. So far,
> > > > > I've managed to integrate Xen and Linux into an existing product
> > > > > that exclusively runs bare-metal code on a ZynqMP SoC and
> > > > > migrate some of the functionality into custom Linux driver/userspace.
> > > > >
> > > > > I'm now looking at low power support, for now at least between
> > > > > Xen
> > > > > (4.16) and Linux (5.15) dom0. I've tried a few different Linux
> > > > > kernel configs around power management and each time I try to
> > > > > suspend from linux dom0 (via sysfs or systemctl), Xen will
> > > > > watchdog on
> > > > > dom0 guest.
> > > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what I
> > > > > can tell debugging through the linux suspend process is it's
> > > > > spinning in a 'suspend- to-idle' loop before it can get to
> > > > > issuing a 'WFI' or using PSCI interface to notify Xen. I'm
> > > > > beginning to suspect that 'low power' support for embedded arm64
> > > > > just isn't quite there yet, or am I missing something in the configs?
> > > > >
> > > > > I realize this could very well be a Linux 'issue' but checking
> > > > > here first. I know Xen presents a flattened device tree to
> > > > > Linux without CPU idle-state nodes and maybe this is causing the
> > > > > linux guest to only do the suspend- to-idle mode? I should
> > > > > mention that I'm booting up using dom0less feature if that
> > > > > matters.
> > > >
> > > >
> > > > Hi Anthony,
> > > >
> > > > Assuming you are using the default Xen command line parameters for
> > > > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI,
> > > > the CPU will execute WFI directly and go into low power mode.
> > > Yes, using these command line params.
> > >
> > > > Given the issue you are describing, I am suspecting the guest is
> > > > not issuing
> > > > WFI: that is simple and known to work. Instead, I suspect that
> > > > Linux might be trying to use PSCI_suspend in a way that is not
> > > > supported or well- implemented by Xen.
> > > >
> > > > Can you check? You can add a printk in Linux
> > > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > > Instrumented both places it doesn't appear to reach there. In
> > > kernel/power/suspend.c, there's a call to s2idle_loop that it's currently
> > > 'stuck'
> > > in and I think it doesn't get to the psci suspend your referring
> > > till afterwards, when suspend_ops->enter is called. Unfortunately,
> > > without any idle-states nodes in the FDT, the only suspend state
> > > Linux is defaults to is 'suspend to idle'.
> >
> > The fact that Linux uses "suspend to idle" is not a problem because as
> > I mentioned WFI or PSCI_suspent are not different on Xen. That part is OK.
> What if using "suspend to idle" is preventing a WFI/PSCI_suspend? Which is
> what I believe I'm currently seeing in my setup. In kernel/power/suspend.c,
> suspend_devices_and_enter(), it gets into the this s2idle_loop and upon
> resuming from idle, it jumps past the point where I believe a
> WFI/PSCI_suspend can happen.
> if (state == PM_SUSPEND_TO_IDLE) {
> s2idle_loop();
> goto Platform_wake;
> }
If that is the case, then it looks like a Linux bug. Maybe something
along these lines?
https://lore.kernel.org/linux-arm-kernel/4665489.GXAFRqVoOG@kreacher/T/#m6edda92d0b5dc09f8e05e7d6db3807501b7249f4
RE: [EXT] Re: xen arm64 low power sleep support
On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> On Wed, 30 Aug 2023, Anthony Chan wrote:
> > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > Hi all,
> > > >
> > > > My name is Tony and I've been researching/developing using Xen
> > > > for potential upcoming uses in our embedded systems. I started
> > > > with Xen using Xilinx tools about a year ago and still have lots
> > > > to learn about what it can to do in the embedded space. So far,
> > > > I've managed to integrate Xen and Linux into an existing product
> > > > that exclusively runs bare-metal code on a ZynqMP SoC and
> > > > migrate some of the functionality into custom Linux driver/userspace.
> > > >
> > > > I'm now looking at low power support, for now at least between
> > > > Xen
> > > > (4.16) and Linux (5.15) dom0. I've tried a few different Linux
> > > > kernel configs around power management and each time I try to
> > > > suspend from linux dom0 (via sysfs or systemctl), Xen will
> > > > watchdog on
> > > > dom0 guest.
> > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what I
> > > > can tell debugging through the linux suspend process is it's
> > > > spinning in a 'suspend- to-idle' loop before it can get to
> > > > issuing a 'WFI' or using PSCI interface to notify Xen. I'm
> > > > beginning to suspect that 'low power' support for embedded arm64
> > > > just isn't quite there yet, or am I missing something in the configs?
> > > >
> > > > I realize this could very well be a Linux 'issue' but checking
> > > > here first. I know Xen presents a flattened device tree to
> > > > Linux without CPU idle-state nodes and maybe this is causing the
> > > > linux guest to only do the suspend- to-idle mode? I should
> > > > mention that I'm booting up using dom0less feature if that
> > > > matters.
> > >
> > >
> > > Hi Anthony,
> > >
> > > Assuming you are using the default Xen command line parameters for
> > > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI,
> > > the CPU will execute WFI directly and go into low power mode.
> > Yes, using these command line params.
> >
> > > Given the issue you are describing, I am suspecting the guest is
> > > not issuing
> > > WFI: that is simple and known to work. Instead, I suspect that
> > > Linux might be trying to use PSCI_suspend in a way that is not
> > > supported or well- implemented by Xen.
> > >
> > > Can you check? You can add a printk in Linux
> > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > Instrumented both places it doesn't appear to reach there. In
> > kernel/power/suspend.c, there's a call to s2idle_loop that it's currently
> > 'stuck'
> > in and I think it doesn't get to the psci suspend your referring
> > till afterwards, when suspend_ops->enter is called. Unfortunately,
> > without any idle-states nodes in the FDT, the only suspend state
> > Linux is defaults to is 'suspend to idle'.
>
> The fact that Linux uses "suspend to idle" is not a problem because as
> I mentioned WFI or PSCI_suspent are not different on Xen. That part is OK.
What if using "suspend to idle" is preventing a WFI/PSCI_suspend? Which is
what I believe I'm currently seeing in my setup. In kernel/power/suspend.c,
suspend_devices_and_enter(), it gets into the this s2idle_loop and upon
resuming from idle, it jumps past the point where I believe a
WFI/PSCI_suspend can happen.
if (state == PM_SUSPEND_TO_IDLE) {
s2idle_loop();
goto Platform_wake;
}
> However, if the issue is not PSCI_suspend then I don't have another
> easy guess. Please post a full stack trace or more information about
> the error in Linux and I might be able to see where it is coming from.
echo mem > /sys/power/state
[ 75.666385] PM: suspend entry (s2idle)
[ 75.685382] Filesystems sync: 0.018 seconds
[ 75.685446] Preparing system for sleep (s2idle)
[ 75.695087] Freezing user space processes ... (elapsed 0.001 seconds) done.
[ 75.698339] OOM killer disabled.
[ 75.701618] Freezing remaining freezable tasks ... (elapsed 0.063 seconds)
done.
[ 75.767215] Suspending system (s2idle)
[ 75.777682] macb ff0e.ethernet: gem-ptp-timer ptp clock unregistered.
[ 75.784525] suspend-to-idle
[ 75.784580] CPU: 0 PID: 548 Comm: sh Not tainted 5.15.0-xilinx-v2022.1 #1
[ 75.788626] Hardware name: xlnx,zynqmp (DT)
[ 75.792867] Call trace:
[ 75.795373] dump_backtrace+0x0/0x1a8
[ 75.799094] show_stack+0x18/0x30
[ 75.802466] dump_stack_lvl+0x7c/0xa0
[ 75.806186] dump_stack+0x18/0x34
[ 75.809559] suspend_devices_and_enter+0x334/0x4cc
[ 75.814403] pm_suspend+0x1f0/0x248
[ 75.817950] state_store+0xb8/0x100
[ 75.821497] kobj_attr_store+0x1c/0x30
[ 75.825302] sysfs_kf_write+0x40/0x54
[ 75.829022] kernfs_fop_write_iter+0xb8/0x170
[ 75.833433] new_sync_write+0x74/0xc8
[ 75.837153] vfs_write+0x104/0x128
[ 75.840613]
RE: [EXT] Re: xen arm64 low power sleep support
On Wed, 30 Aug 2023, Anthony Chan wrote: > On Tue, 29 Aug 2023, Stefano Stabellini wrote: > > On Tue, 29 Aug 2023, Anthony Chan wrote: > > > Hi all, > > > > > > My name is Tony and I've been researching/developing using Xen for > > potential upcoming uses in our embedded systems. I started with Xen > > using Xilinx tools about a year ago and still have lots to learn about what > > it > > can to do in the embedded space. So far, I've managed to integrate Xen > > and Linux into an existing product that exclusively runs bare-metal code on > > a ZynqMP SoC and migrate some of the functionality into custom Linux > > driver/userspace. > > > > > > I'm now looking at low power support, for now at least between Xen > > (4.16) and Linux (5.15) dom0. I've tried a few different Linux kernel > > configs around power management and each time I try to suspend from > > linux dom0 (via sysfs or systemctl), Xen will watchdog on dom0 guest. > > AFAIK, Xen should trap on a 'WFI' from guests, but from what I can tell > > debugging through the linux suspend process is it's spinning in a 'suspend- > > to-idle' loop before it can get to issuing a 'WFI' or using PSCI interface > > to > > notify Xen. I'm beginning to suspect that 'low power' support for > > embedded arm64 just isn't quite there yet, or am I missing something in > > the configs? > > > > > > I realize this could very well be a Linux 'issue' but checking here > > > first. I > > know Xen presents a flattened device tree to Linux without CPU idle-state > > nodes and maybe this is causing the linux guest to only do the suspend- > > to-idle mode? I should mention that I'm booting up using dom0less > > feature if that matters. > > > > > > Hi Anthony, > > > > Assuming you are using the default Xen command line parameters for > > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI, the CPU > > will execute WFI directly and go into low power mode. > Yes, using these command line params. > > > Given the issue you are describing, I am suspecting the guest is not issuing > > WFI: that is simple and known to work. Instead, I suspect that Linux might > > be trying to use PSCI_suspend in a way that is not supported or well- > > implemented by Xen. > > > > Can you check? You can add a printk in Linux > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend > Instrumented both places it doesn't appear to reach there. In > kernel/power/suspend.c, there's a call to s2idle_loop that it's currently > 'stuck' in and I think it doesn't get to the psci suspend your referring till > afterwards, when suspend_ops->enter is called. Unfortunately, without any > idle-states nodes in the FDT, the only suspend state Linux is defaults to is > 'suspend to idle'. The fact that Linux uses "suspend to idle" is not a problem because as I mentioned WFI or PSCI_suspent are not different on Xen. That part is OK. However, if the issue is not PSCI_suspend then I don't have another easy guess. Please post a full stack trace or more information about the error in Linux and I might be able to see where it is coming from.
RE: [EXT] Re: xen arm64 low power sleep support
-Original Message- From: Stefano Stabellini Sent: Tuesday, August 29, 2023 4:13 PM To: Anthony Chan Cc: xen-devel@lists.xenproject.org; bertrand.marq...@arm.com; jul...@xen.org; volodymyr_babc...@epam.com; michal.or...@amd.com; Dan Waqar ; sstabell...@kernel.org Subject: [EXT] Re: xen arm64 low power sleep support On Tue, 29 Aug 2023, Anthony Chan wrote: > Hi all, > > My name is Tony and I've been researching/developing using Xen for potential > upcoming uses in our embedded systems. I started with Xen using Xilinx tools > about a year ago and still have lots to learn about what it can to do in the > embedded space. So far, I've managed to integrate Xen and Linux into an > existing product that exclusively runs bare-metal code on a ZynqMP SoC and > migrate some of the functionality into custom Linux driver/userspace. > > I'm now looking at low power support, for now at least between Xen (4.16) and > Linux (5.15) dom0. I've tried a few different Linux kernel configs around > power management and each time I try to suspend from linux dom0 (via sysfs or > systemctl), Xen will watchdog on dom0 guest. AFAIK, Xen should trap on a > 'WFI' from guests, but from what I can tell debugging through the linux > suspend process is it's spinning in a 'suspend-to-idle' loop before it can > get to issuing a 'WFI' or using PSCI interface to notify Xen. I'm beginning > to suspect that 'low power' support for embedded arm64 just isn't quite there > yet, or am I missing something in the configs? > > I realize this could very well be a Linux 'issue' but checking here first. I > know Xen presents a flattened device tree to Linux without CPU idle-state > nodes and maybe this is causing the linux guest to only do the > suspend-to-idle mode? I should mention that I'm booting up using dom0less > feature if that matters. Hi Anthony, Assuming you are using the default Xen command line parameters for Xilinx boards: sched=null vwfi=native, then if the guest uses WFI, the CPU will execute WFI directly and go into low power mode. Given the issue you are describing, I am suspecting the guest is not issuing WFI: that is simple and known to work. Instead, I suspect that Linux might be trying to use PSCI_suspend in a way that is not supported or well-implemented by Xen. Can you check? You can add a printk in Linux drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend If that is the case, the attached patch might work because it disables PSCI_suspend support in Xen and Linux should fall back to WFI. (There is no power saving in using PSCI_suspend versus WFI.) Cheers, Stefano CONFIDENTIALITY NOTICE: This e-mail, including any attachments, may contain information that is confidential and privileged. Any unauthorized disclosure, reproduction or use of this e-mail is prohibited. If you are not the intended recipient, please notify the sender by reply e-mail or telephone and permanently delete this e-mail and any reproductions immediately.
RE: [EXT] Re: xen arm64 low power sleep support
On Tue, 29 Aug 2023, Stefano Stabellini wrote: > On Tue, 29 Aug 2023, Anthony Chan wrote: > > Hi all, > > > > My name is Tony and I've been researching/developing using Xen for > potential upcoming uses in our embedded systems. I started with Xen > using Xilinx tools about a year ago and still have lots to learn about what it > can to do in the embedded space. So far, I've managed to integrate Xen > and Linux into an existing product that exclusively runs bare-metal code on > a ZynqMP SoC and migrate some of the functionality into custom Linux > driver/userspace. > > > > I'm now looking at low power support, for now at least between Xen > (4.16) and Linux (5.15) dom0. I've tried a few different Linux kernel > configs around power management and each time I try to suspend from > linux dom0 (via sysfs or systemctl), Xen will watchdog on dom0 guest. > AFAIK, Xen should trap on a 'WFI' from guests, but from what I can tell > debugging through the linux suspend process is it's spinning in a 'suspend- > to-idle' loop before it can get to issuing a 'WFI' or using PSCI interface to > notify Xen. I'm beginning to suspect that 'low power' support for > embedded arm64 just isn't quite there yet, or am I missing something in > the configs? > > > > I realize this could very well be a Linux 'issue' but checking here first. > > I > know Xen presents a flattened device tree to Linux without CPU idle-state > nodes and maybe this is causing the linux guest to only do the suspend- > to-idle mode? I should mention that I'm booting up using dom0less > feature if that matters. > > > Hi Anthony, > > Assuming you are using the default Xen command line parameters for > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI, the CPU > will execute WFI directly and go into low power mode. Yes, using these command line params. > Given the issue you are describing, I am suspecting the guest is not issuing > WFI: that is simple and known to work. Instead, I suspect that Linux might > be trying to use PSCI_suspend in a way that is not supported or well- > implemented by Xen. > > Can you check? You can add a printk in Linux > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend Instrumented both places it doesn't appear to reach there. In kernel/power/suspend.c, there's a call to s2idle_loop that it's currently 'stuck' in and I think it doesn't get to the psci suspend your referring till afterwards, when suspend_ops->enter is called. Unfortunately, without any idle-states nodes in the FDT, the only suspend state Linux is defaults to is 'suspend to idle'. Sorry about the boilerplate confidentiality footer below, I am not allowed to disable it... CONFIDENTIALITY NOTICE: This e-mail, including any attachments, may contain information that is confidential and privileged. Any unauthorized disclosure, reproduction or use of this e-mail is prohibited. If you are not the intended recipient, please notify the sender by reply e-mail or telephone and permanently delete this e-mail and any reproductions immediately.
