On Thu, 30 Aug 2023, Stefano Stabellini wrote:
> On Wed, 30 Aug 2023, Anthony Chan wrote:
> > On Tue, 29 Aug 2023, Stefano Stabellini wrote:
> > > On Tue, 29 Aug 2023, Anthony Chan wrote:
> > > > Hi all,
> > > >
> > > > My name is Tony and I've been researching/developing using Xen
> > > > for potential upcoming uses in our embedded systems. I started
> > > > with Xen using Xilinx tools about a year ago and still have lots
> > > > to learn about what it can to do in the embedded space. So far,
> > > > I've managed to integrate Xen and Linux into an existing product
> > > > that exclusively runs bare-metal code on a ZynqMP SoC and
> > > > migrate some of the functionality into custom Linux driver/userspace.
> > > >
> > > > I'm now looking at low power support, for now at least between
> > > > Xen
> > > > (4.16) and Linux (5.15) dom0. I've tried a few different Linux
> > > > kernel configs around power management and each time I try to
> > > > suspend from linux dom0 (via sysfs or systemctl), Xen will
> > > > watchdog on
> > > > dom0 guest.
> > > > AFAIK, Xen should trap on a 'WFI' from guests, but from what I
> > > > can tell debugging through the linux suspend process is it's
> > > > spinning in a 'suspend- to-idle' loop before it can get to
> > > > issuing a 'WFI' or using PSCI interface to notify Xen. I'm
> > > > beginning to suspect that 'low power' support for embedded arm64
> > > > just isn't quite there yet, or am I missing something in the configs?
> > > >
> > > > I realize this could very well be a Linux 'issue' but checking
> > > > here first. I know Xen presents a flattened device tree to
> > > > Linux without CPU idle-state nodes and maybe this is causing the
> > > > linux guest to only do the suspend- to-idle mode? I should
> > > > mention that I'm booting up using dom0less feature if that
> > > > matters.
> > >
> > >
> > > Hi Anthony,
> > >
> > > Assuming you are using the default Xen command line parameters for
> > > Xilinx boards: sched=null vwfi=native, then if the guest uses WFI,
> > > the CPU will execute WFI directly and go into low power mode.
> > Yes, using these command line params.
> >
> > > Given the issue you are describing, I am suspecting the guest is
> > > not issuing
> > > WFI: that is simple and known to work. Instead, I suspect that
> > > Linux might be trying to use PSCI_suspend in a way that is not
> > > supported or well- implemented by Xen.
> > >
> > > Can you check? You can add a printk in Linux
> > > drivers/firmware/psci/psci.c:__psci_cpu_suspend or in Xen
> > > xen/arch/arm/vpsci.c:do_psci_0_2_cpu_suspend
> > Instrumented both places it doesn't appear to reach there. In
> > kernel/power/suspend.c, there's a call to s2idle_loop that it's currently
> > 'stuck'
> > in and I think it doesn't get to the psci suspend your referring
> > till afterwards, when suspend_ops->enter is called. Unfortunately,
> > without any idle-states nodes in the FDT, the only suspend state
> > Linux is defaults to is 'suspend to idle'.
>
> The fact that Linux uses "suspend to idle" is not a problem because as
> I mentioned WFI or PSCI_suspent are not different on Xen. That part is OK.
What if using "suspend to idle" is preventing a WFI/PSCI_suspend? Which is
what I believe I'm currently seeing in my setup. In kernel/power/suspend.c,
suspend_devices_and_enter(), it gets into the this s2idle_loop and upon
resuming from idle, it jumps past the point where I believe a
WFI/PSCI_suspend can happen.
if (state == PM_SUSPEND_TO_IDLE) {
s2idle_loop();
goto Platform_wake;
}
> However, if the issue is not PSCI_suspend then I don't have another
> easy guess. Please post a full stack trace or more information about
> the error in Linux and I might be able to see where it is coming from.
echo mem > /sys/power/state
[ 75.666385] PM: suspend entry (s2idle)
[ 75.685382] Filesystems sync: 0.018 seconds
[ 75.685446] Preparing system for sleep (s2idle)
[ 75.695087] Freezing user space processes ... (elapsed 0.001 seconds) done.
[ 75.698339] OOM killer disabled.
[ 75.701618] Freezing remaining freezable tasks ... (elapsed 0.063 seconds)
done.
[ 75.767215] Suspending system (s2idle)
[ 75.777682] macb ff0e0000.ethernet: gem-ptp-timer ptp clock unregistered.
[ 75.784525] suspend-to-idle
[ 75.784580] CPU: 0 PID: 548 Comm: sh Not tainted 5.15.0-xilinx-v2022.1 #1
[ 75.788626] Hardware name: xlnx,zynqmp (DT)
[ 75.792867] Call trace:
[ 75.795373] dump_backtrace+0x0/0x1a8
[ 75.799094] show_stack+0x18/0x30
[ 75.802466] dump_stack_lvl+0x7c/0xa0
[ 75.806186] dump_stack+0x18/0x34
[ 75.809559] suspend_devices_and_enter+0x334/0x4cc
[ 75.814403] pm_suspend+0x1f0/0x248
[ 75.817950] state_store+0xb8/0x100
[ 75.821497] kobj_attr_store+0x1c/0x30
[ 75.825302] sysfs_kf_write+0x40/0x54
[ 75.829022] kernfs_fop_write_iter+0xb8/0x170
[ 75.833433] new_sync_write+0x74/0xc8
[ 75.837153] vfs_write+0x104/0x128
[ 75.840613] ksys_write+0x70/0xdc
[ 75.843987] __arm64_sys_write+0x18/0x24
[ 75.847965] invoke_syscall+0x74/0xfc
[ 75.851684] el0_svc_common.constprop.0+0xb0/0xdc
[ 75.856442] do_el0_svc+0x84/0x90
[ 75.859816] el0_svc+0x20/0x4c
[ 75.862930] el0t_64_sync_handler+0xbc/0x158
[ 75.867254] el0t_64_sync+0x1a0/0x1a4
CONFIDENTIALITY NOTICE: This e-mail, including any attachments, may contain
information that is confidential and privileged. Any unauthorized disclosure,
reproduction or use of this e-mail is prohibited. If you are not the intended
recipient, please notify the sender by reply e-mail or telephone and
permanently delete this e-mail and any reproductions immediately.
