On 3/19/20 7:08 AM, Haren Myneni wrote: > > On power9, Virtual Accelerator Switchboard (VAS) allows user space or > kernel to communicate with Nest Accelerator (NX) directly using COPY/PASTE > instructions. NX provides various functionalities such as compression, > encryption and etc. But only compression (842 and GZIP formats) is > supported in Linux kernel on power9. > > 842 compression driver (drivers/crypto/nx/nx-842-powernv.c) > is already included in Linux. Only GZIP support will be available from > user space. > > Applications can issue GZIP compression / decompression requests to NX with > COPY/PASTE instructions. When NX is processing these requests, can hit > fault on the request buffer (not in memory). It issues an interrupt and > pastes fault CRB in fault FIFO. Expects kernel to handle this fault and > return credits for both send and fault windows after processing. > > This patch series adds IRQ and fault window setup, and NX fault handling: > - Alloc IRQ and trigger port address, and configure IRQ per VAS instance.
Is the model similar to OCXL ? If so, I suppose that the IRQ for fault handling is allocated by skiboot, exposed in the DT and automatically mapped in Linux when the driver is loaded. Are there other interrupts ? Such as for job completion ? Thanks, C. > - Set port# for each window to generate an interrupt when noticed fault. > - Set fault window and FIFO on which NX paste fault CRB. > - Setup IRQ thread fault handler per VAS instance. > - When receiving an interrupt, Read CRBs from fault FIFO and update > coprocessor_status_block (CSB) in the corresponding CRB with translation > failure (CSB_CC_TRANSLATION). After issuing NX requests, process polls > on CSB address. When it sees translation error, can touch the request > buffer to bring the page in to memory and reissue NX request. > - If copy_to_user fails on user space CSB address, OS sends SEGV signal. > > Tested these patches with NX-GZIP support and will be posting this series > soon. > > Patches 1 & 2: Define alloc IRQ and get port address per chip which are needed > to alloc IRQ per VAS instance. > Patch 3: Define nx_fault_stamp on which NX writes fault status for the fault > CRB > Patch 4: Alloc and setup IRQ and trigger port address for each VAS instance > Patch 5: Setup fault window per each VAS instance. This window is used for > NX to paste fault CRB in FIFO. > Patches 6 & 7: Setup threaded IRQ per VAS and register NX with fault window > ID and port number for each send window so that NX paste fault CRB > in this window. > Patch 8: Reference to pid and mm so that pid is not used until window closed. > Needed for multi thread application where child can open a window > and can be used by parent later. > Patches 9 and 10: Process CRBs from fault FIFO and notify tasks by > updating CSB or through signals. > Patches 11 and 12: Return credits for send and fault windows after handling > faults. > Patch 14:Fix closing send window after all credits are returned. This issue > happens only for user space requests. No page faults on kernel > request buffer. > > Changelog: > V2: > - Use threaded IRQ instead of own kernel thread handler > - Use pswid instead of user space CSB address to find valid CRB > - Removed unused macros and other changes as suggested by Christoph Hellwig > > V3: > - Rebased to 5.5-rc2 > - Use struct pid * instead of pid_t for vas_window tgid > - Code cleanup as suggested by Christoph Hellwig > > V4: > - Define xive alloc and get IRQ info based on chip ID and use these > functions for IRQ setup per VAS instance. It eliminates skiboot > dependency as suggested by Oliver. > > V5: > - Do not update CSB if the process is exiting (patch9) > > V6: > - Add interrupt handler instead of default one and return IRQ_HANDLED > if the fault handling thread is already in progress. (Patch6) > - Use platform send window ID and CCW[0] bit to find valid CRB in > fault FIFO (Patch6). > - Return fault address to user space in BE and other changes as > suggested by Michael Neuling. (patch9) > - Rebased to 5.6-rc4 > > V7: > - Fix sparse warnings (patches 6,9 and 10) > > V8: > - Move mm_context_remove_copro() before mmdrop() (patch8) > - Move barrier before csb.flags store and add WARN_ON_ONCE() checks (patch9) > > Haren Myneni (14): > powerpc/xive: Define xive_native_alloc_irq_on_chip() > powerpc/xive: Define xive_native_alloc_get_irq_info() > powerpc/vas: Define nx_fault_stamp in coprocessor_request_block > powerpc/vas: Alloc and setup IRQ and trigger port address > powerpc/vas: Setup fault window per VAS instance > powerpc/vas: Setup thread IRQ handler per VAS instance > powerpc/vas: Register NX with fault window ID and IRQ port value > powerpc/vas: Take reference to PID and mm for user space windows > powerpc/vas: Update CSB and notify process for fault CRBs > powerpc/vas: Print CRB and FIFO values > powerpc/vas: Do not use default credits for receive window > powerpc/vas: Return credits after handling fault > powerpc/vas: Display process stuck message > powerpc/vas: Free send window in VAS instance after credits returned > > arch/powerpc/include/asm/icswx.h | 18 +- > arch/powerpc/include/asm/xive.h | 11 +- > arch/powerpc/platforms/powernv/Makefile | 2 +- > arch/powerpc/platforms/powernv/ocxl.c | 20 +- > arch/powerpc/platforms/powernv/vas-debug.c | 2 +- > arch/powerpc/platforms/powernv/vas-fault.c | 332 > ++++++++++++++++++++++++++++ > arch/powerpc/platforms/powernv/vas-window.c | 185 ++++++++++++++-- > arch/powerpc/platforms/powernv/vas.c | 101 ++++++++- > arch/powerpc/platforms/powernv/vas.h | 51 ++++- > arch/powerpc/sysdev/xive/native.c | 29 ++- > 10 files changed, 704 insertions(+), 47 deletions(-) > create mode 100644 arch/powerpc/platforms/powernv/vas-fault.c >