Re: [PATCH v3] powerpc/powernv: Enable PCI peer-to-peer

[Michael Ellerman]

Brian King  writes:

> Michael,
>
> What do we need on this one before we can pull into your -next branch?

This skiboot side to be merged.

cheers

Re: [PATCH v3] powerpc/powernv: Enable PCI peer-to-peer

[Brian King]

Michael,

What do we need on this one before we can pull into your -next branch?

Thanks,

Brian

-- 
Brian King
Power Linux I/O
IBM Linux Technology Center

[PATCH v3] powerpc/powernv: Enable PCI peer-to-peer

[Frederic Barrat]

P9 has support for PCI peer-to-peer, enabling a device to write in the
mmio space of another device directly, without interrupting the CPU.

This patch adds support for it on powernv, by adding a new API to be
called by drivers. The pnv_pci_set_p2p(...) call configures an
'initiator', i.e the device which will issue the mmio operation, and a
'target', i.e. the device on the receiving side.

P9 really only supports mmio stores for the time being but that's
expected to change in the future, so the API allows to define both
load and store operations.

/* PCI p2p descriptor */
#define OPAL_PCI_P2P_ENABLE 0x1
#define OPAL_PCI_P2P_LOAD   0x2
#define OPAL_PCI_P2P_STORE  0x4

int pnv_pci_set_p2p(struct pci_dev *initiator, struct pci_dev *target,
   uint64_t desc)

It uses a new OPAL call, as the configuration magic is done on the
PHBs by skiboot.

Signed-off-by: Frederic Barrat 
---
Changelog:
Requires skiboot patch:
http://patchwork.ozlabs.org/patch/789763/

Changelog:
v3:
  - move target reference count from skiboot to linux
v2:
  - change of API
  - allow disabling of p2p setting
  
 arch/powerpc/include/asm/opal-api.h|  8 ++-
 arch/powerpc/include/asm/opal.h|  2 +
 arch/powerpc/include/asm/pnv-pci.h |  2 +
 arch/powerpc/platforms/powernv/opal-wrappers.S |  1 +
 arch/powerpc/platforms/powernv/pci-ioda.c  |  3 +-
 arch/powerpc/platforms/powernv/pci.c   | 71 ++
 arch/powerpc/platforms/powernv/pci.h   |  5 ++
 7 files changed, 89 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/include/asm/opal-api.h 
b/arch/powerpc/include/asm/opal-api.h
index 3130a73652c7..d7ae1bb1cabd 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -190,7 +190,8 @@
 #define OPAL_NPU_INIT_CONTEXT  146
 #define OPAL_NPU_DESTROY_CONTEXT   147
 #define OPAL_NPU_MAP_LPAR  148
-#define OPAL_LAST  148
+#define OPAL_PCI_SET_P2P   152
+#define OPAL_LAST  152
 
 /* Device tree flags */
 
@@ -1084,6 +1085,11 @@ enum {
XIVE_DUMP_EMU_STATE = 5,
 };
 
+/* PCI p2p descriptor */
+#define OPAL_PCI_P2P_ENABLE0x1
+#define OPAL_PCI_P2P_LOAD  0x2
+#define OPAL_PCI_P2P_STORE 0x4
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* __OPAL_API_H */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 588fb1c23af9..bd9674f4c4ae 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -267,6 +267,8 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
 int64_t opal_xive_free_irq(uint32_t girq);
 int64_t opal_xive_sync(uint32_t type, uint32_t id);
 int64_t opal_xive_dump(uint32_t type, uint32_t id);
+int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
+   uint64_t desc, uint16_t pe_number);
 
 /* Internal functions */
 extern int early_init_dt_scan_opal(unsigned long node, const char *uname,
diff --git a/arch/powerpc/include/asm/pnv-pci.h 
b/arch/powerpc/include/asm/pnv-pci.h
index de9681034353..59a548909d0b 100644
--- a/arch/powerpc/include/asm/pnv-pci.h
+++ b/arch/powerpc/include/asm/pnv-pci.h
@@ -26,6 +26,8 @@ extern int pnv_pci_get_presence_state(uint64_t id, uint8_t 
*state);
 extern int pnv_pci_get_power_state(uint64_t id, uint8_t *state);
 extern int pnv_pci_set_power_state(uint64_t id, uint8_t state,
   struct opal_msg *msg);
+extern int pnv_pci_set_p2p(struct pci_dev *initiator, struct pci_dev *target,
+   uint64_t desc);
 
 int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode);
 int pnv_cxl_ioda_msi_setup(struct pci_dev *dev, unsigned int hwirq,
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S 
b/arch/powerpc/platforms/powernv/opal-wrappers.S
index 4ca6c26a56d5..8cc0a6266111 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -310,3 +310,4 @@ OPAL_CALL(opal_xive_dump,   OPAL_XIVE_DUMP);
 OPAL_CALL(opal_npu_init_context,   OPAL_NPU_INIT_CONTEXT);
 OPAL_CALL(opal_npu_destroy_context,OPAL_NPU_DESTROY_CONTEXT);
 OPAL_CALL(opal_npu_map_lpar,   OPAL_NPU_MAP_LPAR);
+OPAL_CALL(opal_pci_set_p2p,OPAL_PCI_SET_P2P);
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c 
b/arch/powerpc/platforms/powernv/pci-ioda.c
index 437613588df1..026a06c51458 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -1408,7 +1408,6 @@ static int pnv_pci_vf_assign_m64(struct pci_dev *pdev, 
u16 num_vfs)
 
 static long pnv_pci_ioda2_unset_window(struct iommu_table_group *table_group,
int num);
-static void pnv_pci_ioda2_set_bypass(struct