PowerPC provides HW compression with NX coprocessor. This feature
is available on both PowerNV and PowerVM and included in Linux.
Since each powerpc chip has one NX coprocessor, the VAS introduces
the concept of windows / credits to manage access to this hardware
resource. On powerVM, these limited resources should be available
across all LPARs. So the hypervisor assigns the specific credits
to each LPAR based on processor entitlement so that one LPAR does
not overload NX. The hypervisor can reject the window open request
to a partition if exceeds its credit limit (1 credit per window).
So the total number of target credits in a partition can be changed
if the core configuration is modified. The hypervisor expects the
partition to modify its window usage depends on new target
credits. For example, if the partition uses more credits than the
new target credits, it should close the excessive windows so that
the NX resource will be available to other partitions.
This patch series enables OS to support this dynamic credit
management with DLPAR core removal/add.
Core removal operation:
- Get new VAS capabilities from the hypervisor when the DLPAR
notifier is received. This capabilities provides the new target
credits based on new processor entitlement. In the case of QoS
credit changes, the notification will be issued by updating
the target_creds via sysfs.
- If the partition is already used more than the new target credits,
the kernel selects windows, unmap the current paste address and
close them in the hypervisor, It uses FIFO to identify these
windows - last windows that are opened are the first ones to be
closed.
- When the user space issue requests on these windows, NX generates
page fault on the unmap paste address. The kernel handles the
fault by returning the paste instruction failure if the window is
not active (means unmap paste). Then up to the library / user
space to fall back to SW compression or manage with the current
windows.
Core add operation:
- The kernel can see increased target credits from the new VAS
capabilities.
- Scans the window list for the closed windows in the hypervisor
due to lost credit before and selects windows based on same FIFO.
- Make these corresponding windows active and create remap with
the same VMA on the new paste address in the fault handler.
- Then the user space should expect paste successful later.
Patch 1: Define common names for sysfs target/used/avail_creds so
that same sysfs entries can be used even on PowerNV later.
Patch 2: Save PID in the vas window struct during initial window
open and use it when reopen later.
Patch 3: Add new mmap fault handler which handles the page fault
from NX on paste address.
Patch 4: Return the paste instruction failure if the window is not
active.
Patch 5: If the window is closed in the hypervisor before the user
space issue the initial mmap(), return -EACCES failure.
Patch 6: Close windows in the hypervisor when the partition exceeds
its usage than the new target credits.
Patch 7: When credits are available, reopen windows that are closed
before with core removal.
Patch 8 & 9: The user space determines the credit usage with sysfs
nr_total/nr_used_credits interfaces. drmgr uses
update_total_credits to notify OS for QoS credit changes.
Thanks to Nicholas Piggin and Aneesh Kumar for the valuable suggestions
on the NXGZIP design to support DLPAR operations.
Changes in v2:
- Rebase on 5.16-rc5
- Use list safe functions to iterate windows list
- Changes to show the actual value in sysfs used_credits even though
some windows are inactive with core removal. Reflects -ve value in
sysfs avail_creds to let userspace know that it opened more windows
than the current maximum LPAR credits.
Changes in v3:
- Rebase on 5.16
- Reconfigure VAS windows only for CPU hotplug events.
Changes in v4:
- Rebase on 5.17-rc4
- Changes based on comments from Nicholas Piggin
- Included VAS DLPAR notifer code in 'Close windows with DLPAR'
patch instead of as a separate patch
- Patches reordering and other changes
Changes in v5:
- Rebase on 5.17-rc5
- Add update_total_credits sysfs entry to update QoS target credits
and other commit descriptions as suggested by Nicholas Piggin
Changed in v6:
- Build fix in "Add paste address mmap fault handler" patch
as reported by kernel test robot <l...@intel.com>
Haren Myneni (9):
powerpc/pseries/vas: Use common names in VAS capability structure
powerpc/pseries/vas: Save PID in pseries_vas_window struct
powerpc/vas: Add paste address mmap fault handler
powerpc/vas: Return paste instruction failure if no active window
powerpc/vas: Map paste address only if window is active
powerpc/pseries/vas: Close windows with DLPAR core removal
powerpc/pseries/vas: Reopen windows with DLPAR core add
powerpc/pseries/vas: sysfs interface to export capabilities
powerpc/pseries/vas: Add 'update_total_credits' entry for QoS
capabilities
arch/powerpc/include/asm/ppc-opcode.h | 2 +
arch/powerpc/include/asm/vas.h | 12 +
arch/powerpc/platforms/book3s/vas-api.c | 145 +++++++++-
arch/powerpc/platforms/pseries/Makefile | 2 +-
arch/powerpc/platforms/pseries/vas-sysfs.c | 268 +++++++++++++++++
arch/powerpc/platforms/pseries/vas.c | 321 +++++++++++++++++++--
arch/powerpc/platforms/pseries/vas.h | 16 +-
7 files changed, 744 insertions(+), 22 deletions(-)
create mode 100644 arch/powerpc/platforms/pseries/vas-sysfs.c
--
2.27.0
