Re: [PATCH v5 14/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge

2020-09-20 Thread Randy Dunlap
On 9/20/20 9:52 PM, Kishon Vijay Abraham I wrote:
> Hi Randy,
> 
> On 18/09/20 9:45 pm, Randy Dunlap wrote:
>> On 9/17/20 11:42 PM, Kishon Vijay Abraham I wrote:
>>> diff --git a/drivers/ntb/hw/epf/Kconfig b/drivers/ntb/hw/epf/Kconfig
>>> new file mode 100644
>>> index ..6197d1aab344
>>> --- /dev/null
>>> +++ b/drivers/ntb/hw/epf/Kconfig
>>> @@ -0,0 +1,6 @@
>>> +config NTB_EPF
>>> +   tristate "Generic EPF Non-Transparent Bridge support"
>>> +   depends on m
>>> +   help
>>> + This driver supports EPF NTB on configurable endpoint.
>>> + If unsure, say N.
>>
>> Hi,
>> Why is this driver restricted to 'm' (loadable module)?
>> I.e., it cannot be builtin.
> 
> I'm trying to keep all the host side PCI drivers corresponding to the
> devices configured using endpoint function drivers as modules and also
> not populate MODULE_DEVICE_TABLE() to prevent auto-loading.
> 
> The different endpoint function drivers (right now only pci-epf-test.c
> and pci-epf-ntb.c) can use the same device ID and vendorID for
> configuring the endpoint devices. So on the host side, it's possible an
> un-intended PCI driver can be bound to the device. So in-order to give
> users the flexibility of deciding the driver to be bound, I'm trying to
> keep it as modules. (Some driver like NTB also uses class code
> PCI_CLASS_MEMORY_RAM for binding a driver in addition to deviceID and
> vendorID but it need not be the case for all the drivers.)

Thanks for the explanation.

-- 
~Randy



Re: [PATCH v5 14/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge

2020-09-20 Thread Kishon Vijay Abraham I
Hi Randy,

On 18/09/20 9:45 pm, Randy Dunlap wrote:
> On 9/17/20 11:42 PM, Kishon Vijay Abraham I wrote:
>> diff --git a/drivers/ntb/hw/epf/Kconfig b/drivers/ntb/hw/epf/Kconfig
>> new file mode 100644
>> index ..6197d1aab344
>> --- /dev/null
>> +++ b/drivers/ntb/hw/epf/Kconfig
>> @@ -0,0 +1,6 @@
>> +config NTB_EPF
>> +tristate "Generic EPF Non-Transparent Bridge support"
>> +depends on m
>> +help
>> +  This driver supports EPF NTB on configurable endpoint.
>> +  If unsure, say N.
> 
> Hi,
> Why is this driver restricted to 'm' (loadable module)?
> I.e., it cannot be builtin.

I'm trying to keep all the host side PCI drivers corresponding to the
devices configured using endpoint function drivers as modules and also
not populate MODULE_DEVICE_TABLE() to prevent auto-loading.

The different endpoint function drivers (right now only pci-epf-test.c
and pci-epf-ntb.c) can use the same device ID and vendorID for
configuring the endpoint devices. So on the host side, it's possible an
un-intended PCI driver can be bound to the device. So in-order to give
users the flexibility of deciding the driver to be bound, I'm trying to
keep it as modules. (Some driver like NTB also uses class code
PCI_CLASS_MEMORY_RAM for binding a driver in addition to deviceID and
vendorID but it need not be the case for all the drivers.)

Thanks
Kishon


Re: [PATCH v5 14/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge

2020-09-18 Thread Randy Dunlap
On 9/17/20 11:42 PM, Kishon Vijay Abraham I wrote:
> diff --git a/drivers/ntb/hw/epf/Kconfig b/drivers/ntb/hw/epf/Kconfig
> new file mode 100644
> index ..6197d1aab344
> --- /dev/null
> +++ b/drivers/ntb/hw/epf/Kconfig
> @@ -0,0 +1,6 @@
> +config NTB_EPF
> + tristate "Generic EPF Non-Transparent Bridge support"
> + depends on m
> + help
> +   This driver supports EPF NTB on configurable endpoint.
> +   If unsure, say N.

Hi,
Why is this driver restricted to 'm' (loadable module)?
I.e., it cannot be builtin.

thanks.
-- 
~Randy



[PATCH v5 14/17] NTB: Add support for EPF PCI-Express Non-Transparent Bridge

2020-09-18 Thread Kishon Vijay Abraham I
Add support for EPF PCI-Express Non-Transparent Bridge (NTB) device.
This driver is platform independent and could be used by any platform
which have multiple PCIe endpoint instances configured using the
pci-epf-ntb driver. The driver connnects to the standard NTB sub-system
interface. The EPF NTB device has configurable number of memory windows
(Max 4), configurable number of doorbell (Max 32), and configurable
number of scratch-pad registers.

Signed-off-by: Kishon Vijay Abraham I 
---
 drivers/ntb/hw/Kconfig  |   1 +
 drivers/ntb/hw/Makefile |   1 +
 drivers/ntb/hw/epf/Kconfig  |   6 +
 drivers/ntb/hw/epf/Makefile |   1 +
 drivers/ntb/hw/epf/ntb_hw_epf.c | 751 
 5 files changed, 760 insertions(+)
 create mode 100644 drivers/ntb/hw/epf/Kconfig
 create mode 100644 drivers/ntb/hw/epf/Makefile
 create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c

diff --git a/drivers/ntb/hw/Kconfig b/drivers/ntb/hw/Kconfig
index e77c587060ff..c325be526b80 100644
--- a/drivers/ntb/hw/Kconfig
+++ b/drivers/ntb/hw/Kconfig
@@ -2,4 +2,5 @@
 source "drivers/ntb/hw/amd/Kconfig"
 source "drivers/ntb/hw/idt/Kconfig"
 source "drivers/ntb/hw/intel/Kconfig"
+source "drivers/ntb/hw/epf/Kconfig"
 source "drivers/ntb/hw/mscc/Kconfig"
diff --git a/drivers/ntb/hw/Makefile b/drivers/ntb/hw/Makefile
index 4714d6238845..223ca592b5f9 100644
--- a/drivers/ntb/hw/Makefile
+++ b/drivers/ntb/hw/Makefile
@@ -2,4 +2,5 @@
 obj-$(CONFIG_NTB_AMD)  += amd/
 obj-$(CONFIG_NTB_IDT)  += idt/
 obj-$(CONFIG_NTB_INTEL)+= intel/
+obj-$(CONFIG_NTB_EPF)  += epf/
 obj-$(CONFIG_NTB_SWITCHTEC) += mscc/
diff --git a/drivers/ntb/hw/epf/Kconfig b/drivers/ntb/hw/epf/Kconfig
new file mode 100644
index ..6197d1aab344
--- /dev/null
+++ b/drivers/ntb/hw/epf/Kconfig
@@ -0,0 +1,6 @@
+config NTB_EPF
+   tristate "Generic EPF Non-Transparent Bridge support"
+   depends on m
+   help
+ This driver supports EPF NTB on configurable endpoint.
+ If unsure, say N.
diff --git a/drivers/ntb/hw/epf/Makefile b/drivers/ntb/hw/epf/Makefile
new file mode 100644
index ..2f560a422bc6
--- /dev/null
+++ b/drivers/ntb/hw/epf/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_NTB_EPF) += ntb_hw_epf.o
diff --git a/drivers/ntb/hw/epf/ntb_hw_epf.c b/drivers/ntb/hw/epf/ntb_hw_epf.c
new file mode 100644
index ..2989f2b41fda
--- /dev/null
+++ b/drivers/ntb/hw/epf/ntb_hw_epf.c
@@ -0,0 +1,751 @@
+// SPDX-License-Identifier: GPL-2.0
+/**
+ * Host side endpoint driver to implement Non-Transparent Bridge functionality
+ *
+ * Copyright (C) 2020 Texas Instruments
+ * Author: Kishon Vijay Abraham I 
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#define NTB_EPF_COMMAND0x0
+#define CMD_CONFIGURE_DOORBELL 1
+#define CMD_TEARDOWN_DOORBELL  2
+#define CMD_CONFIGURE_MW   3
+#define CMD_TEARDOWN_MW4
+#define CMD_LINK_UP5
+#define CMD_LINK_DOWN  6
+
+#define NTB_EPF_ARGUMENT   0x4
+#define MSIX_ENABLEBIT(16)
+
+#define NTB_EPF_CMD_STATUS 0x8
+#define COMMAND_STATUS_OK  1
+#define COMMAND_STATUS_ERROR   2
+
+#define NTB_EPF_LINK_STATUS0x0A
+#define LINK_STATUS_UP BIT(0)
+
+#define NTB_EPF_TOPOLOGY   0x0C
+#define NTB_EPF_LOWER_ADDR 0x10
+#define NTB_EPF_UPPER_ADDR 0x14
+#define NTB_EPF_LOWER_SIZE 0x18
+#define NTB_EPF_UPPER_SIZE 0x1C
+#define NTB_EPF_MW_COUNT   0x20
+#define NTB_EPF_MW1_OFFSET 0x24
+#define NTB_EPF_SPAD_OFFSET0x28
+#define NTB_EPF_SPAD_COUNT 0x2C
+#define NTB_EPF_DB_ENTRY_SIZE  0x30
+#define NTB_EPF_DB_DATA(n) (0x34 + (n) * 4)
+
+#define NTB_EPF_MIN_DB_COUNT   3
+#define NTB_EPF_MAX_DB_COUNT   31
+#define NTB_EPF_MW_OFFSET  2
+
+#define NTB_EPF_COMMAND_TIMEOUT1000 /* 1 Sec */
+
+enum pci_barno {
+   BAR_0,
+   BAR_1,
+   BAR_2,
+   BAR_3,
+   BAR_4,
+   BAR_5,
+};
+
+struct ntb_epf_dev {
+   struct ntb_dev ntb;
+   struct device *dev;
+   /* Mutex to protect providing commands to NTB EPF */
+   struct mutex cmd_lock;
+
+   enum pci_barno ctrl_reg_bar;
+   enum pci_barno peer_spad_reg_bar;
+   enum pci_barno db_reg_bar;
+
+   unsigned int mw_count;
+   unsigned int spad_count;
+   unsigned int db_count;
+
+   void __iomem *ctrl_reg;
+   void __iomem *db_reg;
+   void __iomem *peer_spad_reg;
+
+   unsigned int self_spad;
+   unsigned int peer_spad;
+
+   int db_val;
+   u64 db_valid_mask;
+};
+
+#define ntb_ndev(__ntb) container_of(__ntb, struct ntb_epf_dev, ntb)
+
+struct ntb_epf_data {
+   /* BAR that contains both control region and self spad region */
+   enum pci_barno ctrl_reg_bar;
+   /* BAR that contains peer spad region */
+   enum pci_barno peer_spad_reg_bar;
+   /* BAR that contains Doorbell region and Memory window '1' */
+   enum pci_barno db_reg_bar;
+};
+
+static int ntb_epf_send_command(struct