Re: [PATCH v5] NTB: Add IDT 89HPESxNTx PCIe-switches support

[Jon Mason]

On Tue, Mar 07, 2017 at 05:02:38AM +0300, Serge Semin wrote:
> IDT 89HPESxNTx device series is PCIe-switches, which support
> Non-Transparent bridging between domains connected to the device ports.
> Since new NTB API exposes multi-port interface and messaging API, the
> IDT NT-functions can be now supported in the kernel. This driver adds
> the following functionality:
> 1) Multi-port NTB API to have information of possible NT-functions
> activated in compliance with available device ports.
> 2) Memory windows of direct and look up table based address translation
> with all possible combinations of BARs setup.
> 3) Traditional doorbell NTB API.
> 4) One-on-one messaging NTB API.
> 
> There are some IDT PCIe-switch setups, which must be done before any of
> the NTB peers started. It can be performed either by system BIOS via
> IDT SMBus-slave interface or by pre-initialized IDT PCIe-switch EEPROM:
> 1) NT-functions of corresponding ports must be activated using
> SWPARTxCTL and SWPORTxCTL registers.
> 2) BAR0 must be configured to expose NT-function configuration
> registers map.
> 3) The rest of the BARs must have at least one memory window
> configured, otherwise the driver will just return an error.
> Temperature sensor of IDT PCIe-switches can be also optionally
> activated by BIOS or EEPROM.
> (See IDT documentations for details of how the pre-initialization can
> be done)
> 
> Signed-off-by: Serge Semin 
> Acked-by: Allen Hubbe 
> 
> ---
> 
> Changelog v2:
> - Fix minor checkpatch.pl issues
> - Get rid of obfuscating macros
> 
> Changelog v3:
> - No write to registers if address is either out of bound or unaligned
> - Fix idt_reg_set_bits()/idt_reg_clear_bits() methods race condition
> - Fix invalid argument of write method called from
> idt_reg_set_bits()/idt_reg_clear_bits() functions
> - Add appropriate naming of function idt_get_mw_size()
> - Fix some documentation notes
> - Replace symbolic permission S_IRUSR with octal 0400
> 
> Changelog v4:
> - Return ~0 on read from registers with invalid address
> - Don't check bits validity on registers bits clearing
> - Keep up driver loading (just print a warning) if there is no any peer
> NTBs found
> - Fix unnecessary branching logic
> - Fix some documentation notes
> 
> Changelog v5:
> - Fix minor documentation issues
> - Replace writel/readl with iowrite32/ioread32 methods
> - Discard dev_*() wrappers with origins
> - Use pci_alloc_irq_vectors() for IRQ number and ISR initialization
> - Use Mananged Device Resource as much as possible:
> devm_request_threaded_irq(), pcim_iomap_regions_request_all()
> 
>  drivers/ntb/hw/Kconfig  |1 +
>  drivers/ntb/hw/Makefile |1 +
>  drivers/ntb/hw/idt/Kconfig  |   31 +
>  drivers/ntb/hw/idt/Makefile |1 +
>  drivers/ntb/hw/idt/ntb_hw_idt.c | 2600 
> +++
>  drivers/ntb/hw/idt/ntb_hw_idt.h | 1149 +
>  6 files changed, 3783 insertions(+)
>  create mode 100644 drivers/ntb/hw/idt/Kconfig
>  create mode 100644 drivers/ntb/hw/idt/Makefile
>  create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.c
>  create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.h
> 
> diff --git a/drivers/ntb/hw/Kconfig b/drivers/ntb/hw/Kconfig
> index 7116472..a89243c 100644
> --- a/drivers/ntb/hw/Kconfig
> +++ b/drivers/ntb/hw/Kconfig
> @@ -1,2 +1,3 @@
>  source "drivers/ntb/hw/amd/Kconfig"
> +source "drivers/ntb/hw/idt/Kconfig"
>  source "drivers/ntb/hw/intel/Kconfig"
> diff --git a/drivers/ntb/hw/Makefile b/drivers/ntb/hw/Makefile
> index 532e085..87332c3 100644
> --- a/drivers/ntb/hw/Makefile
> +++ b/drivers/ntb/hw/Makefile
> @@ -1,2 +1,3 @@
>  obj-$(CONFIG_NTB_AMD)+= amd/
> +obj-$(CONFIG_NTB_IDT)+= idt/
>  obj-$(CONFIG_NTB_INTEL)  += intel/
> diff --git a/drivers/ntb/hw/idt/Kconfig b/drivers/ntb/hw/idt/Kconfig
> new file mode 100644
> index 000..b360e56
> --- /dev/null
> +++ b/drivers/ntb/hw/idt/Kconfig
> @@ -0,0 +1,31 @@
> +config NTB_IDT
> + tristate "IDT PCIe-switch Non-Transparent Bridge support"
> + depends on PCI
> + help
> +  This driver supports NTB of cappable IDT PCIe-switches.
> +
> +  Some of the pre-initializations must be made before IDT PCIe-switch
> +  exposes it NT-functions correctly. It should be done by either proper
> +  initialisation of EEPROM connected to master smbus of the switch or
> +  by BIOS using slave-SMBus interface changing corresponding registers
> +  value. Evidently it must be done before PCI bus enumeration is
> +  finished in Linux kernel.
> +
> +  First of all partitions must be activated and properly assigned to all
> +  the ports with NT-functions intended to be activated (see SWPARTxCTL
> +  and SWPORTxCTL registers). Then all NT-function BARs must be enabled
> +  with chosen valid aperture. For memory windows related BARs the
> +  aperture settings shall determine the maximum size of memory 

Re: [PATCH v5] NTB: Add IDT 89HPESxNTx PCIe-switches support

[Jon Mason]

On Tue, Mar 07, 2017 at 05:02:38AM +0300, Serge Semin wrote:
> IDT 89HPESxNTx device series is PCIe-switches, which support
> Non-Transparent bridging between domains connected to the device ports.
> Since new NTB API exposes multi-port interface and messaging API, the
> IDT NT-functions can be now supported in the kernel. This driver adds
> the following functionality:
> 1) Multi-port NTB API to have information of possible NT-functions
> activated in compliance with available device ports.
> 2) Memory windows of direct and look up table based address translation
> with all possible combinations of BARs setup.
> 3) Traditional doorbell NTB API.
> 4) One-on-one messaging NTB API.
> 
> There are some IDT PCIe-switch setups, which must be done before any of
> the NTB peers started. It can be performed either by system BIOS via
> IDT SMBus-slave interface or by pre-initialized IDT PCIe-switch EEPROM:
> 1) NT-functions of corresponding ports must be activated using
> SWPARTxCTL and SWPORTxCTL registers.
> 2) BAR0 must be configured to expose NT-function configuration
> registers map.
> 3) The rest of the BARs must have at least one memory window
> configured, otherwise the driver will just return an error.
> Temperature sensor of IDT PCIe-switches can be also optionally
> activated by BIOS or EEPROM.
> (See IDT documentations for details of how the pre-initialization can
> be done)
> 
> Signed-off-by: Serge Semin 
> Acked-by: Allen Hubbe 
> 
> ---
> 
> Changelog v2:
> - Fix minor checkpatch.pl issues
> - Get rid of obfuscating macros
> 
> Changelog v3:
> - No write to registers if address is either out of bound or unaligned
> - Fix idt_reg_set_bits()/idt_reg_clear_bits() methods race condition
> - Fix invalid argument of write method called from
> idt_reg_set_bits()/idt_reg_clear_bits() functions
> - Add appropriate naming of function idt_get_mw_size()
> - Fix some documentation notes
> - Replace symbolic permission S_IRUSR with octal 0400
> 
> Changelog v4:
> - Return ~0 on read from registers with invalid address
> - Don't check bits validity on registers bits clearing
> - Keep up driver loading (just print a warning) if there is no any peer
> NTBs found
> - Fix unnecessary branching logic
> - Fix some documentation notes
> 
> Changelog v5:
> - Fix minor documentation issues
> - Replace writel/readl with iowrite32/ioread32 methods
> - Discard dev_*() wrappers with origins
> - Use pci_alloc_irq_vectors() for IRQ number and ISR initialization
> - Use Mananged Device Resource as much as possible:
> devm_request_threaded_irq(), pcim_iomap_regions_request_all()
> 
>  drivers/ntb/hw/Kconfig  |1 +
>  drivers/ntb/hw/Makefile |1 +
>  drivers/ntb/hw/idt/Kconfig  |   31 +
>  drivers/ntb/hw/idt/Makefile |1 +
>  drivers/ntb/hw/idt/ntb_hw_idt.c | 2600 
> +++
>  drivers/ntb/hw/idt/ntb_hw_idt.h | 1149 +
>  6 files changed, 3783 insertions(+)
>  create mode 100644 drivers/ntb/hw/idt/Kconfig
>  create mode 100644 drivers/ntb/hw/idt/Makefile
>  create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.c
>  create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.h
> 
> diff --git a/drivers/ntb/hw/Kconfig b/drivers/ntb/hw/Kconfig
> index 7116472..a89243c 100644
> --- a/drivers/ntb/hw/Kconfig
> +++ b/drivers/ntb/hw/Kconfig
> @@ -1,2 +1,3 @@
>  source "drivers/ntb/hw/amd/Kconfig"
> +source "drivers/ntb/hw/idt/Kconfig"
>  source "drivers/ntb/hw/intel/Kconfig"
> diff --git a/drivers/ntb/hw/Makefile b/drivers/ntb/hw/Makefile
> index 532e085..87332c3 100644
> --- a/drivers/ntb/hw/Makefile
> +++ b/drivers/ntb/hw/Makefile
> @@ -1,2 +1,3 @@
>  obj-$(CONFIG_NTB_AMD)+= amd/
> +obj-$(CONFIG_NTB_IDT)+= idt/
>  obj-$(CONFIG_NTB_INTEL)  += intel/
> diff --git a/drivers/ntb/hw/idt/Kconfig b/drivers/ntb/hw/idt/Kconfig
> new file mode 100644
> index 000..b360e56
> --- /dev/null
> +++ b/drivers/ntb/hw/idt/Kconfig
> @@ -0,0 +1,31 @@
> +config NTB_IDT
> + tristate "IDT PCIe-switch Non-Transparent Bridge support"
> + depends on PCI
> + help
> +  This driver supports NTB of cappable IDT PCIe-switches.
> +
> +  Some of the pre-initializations must be made before IDT PCIe-switch
> +  exposes it NT-functions correctly. It should be done by either proper
> +  initialisation of EEPROM connected to master smbus of the switch or
> +  by BIOS using slave-SMBus interface changing corresponding registers
> +  value. Evidently it must be done before PCI bus enumeration is
> +  finished in Linux kernel.
> +
> +  First of all partitions must be activated and properly assigned to all
> +  the ports with NT-functions intended to be activated (see SWPARTxCTL
> +  and SWPORTxCTL registers). Then all NT-function BARs must be enabled
> +  with chosen valid aperture. For memory windows related BARs the
> +  aperture settings shall determine the maximum size of memory windows
> +  accepted by a BAR. Note that BAR0 

[PATCH v5] NTB: Add IDT 89HPESxNTx PCIe-switches support

[Serge Semin]

IDT 89HPESxNTx device series is PCIe-switches, which support
Non-Transparent bridging between domains connected to the device ports.
Since new NTB API exposes multi-port interface and messaging API, the
IDT NT-functions can be now supported in the kernel. This driver adds
the following functionality:
1) Multi-port NTB API to have information of possible NT-functions
activated in compliance with available device ports.
2) Memory windows of direct and look up table based address translation
with all possible combinations of BARs setup.
3) Traditional doorbell NTB API.
4) One-on-one messaging NTB API.

There are some IDT PCIe-switch setups, which must be done before any of
the NTB peers started. It can be performed either by system BIOS via
IDT SMBus-slave interface or by pre-initialized IDT PCIe-switch EEPROM:
1) NT-functions of corresponding ports must be activated using
SWPARTxCTL and SWPORTxCTL registers.
2) BAR0 must be configured to expose NT-function configuration
registers map.
3) The rest of the BARs must have at least one memory window
configured, otherwise the driver will just return an error.
Temperature sensor of IDT PCIe-switches can be also optionally
activated by BIOS or EEPROM.
(See IDT documentations for details of how the pre-initialization can
be done)

Signed-off-by: Serge Semin 
Acked-by: Allen Hubbe 

---

Changelog v2:
- Fix minor checkpatch.pl issues
- Get rid of obfuscating macros

Changelog v3:
- No write to registers if address is either out of bound or unaligned
- Fix idt_reg_set_bits()/idt_reg_clear_bits() methods race condition
- Fix invalid argument of write method called from
idt_reg_set_bits()/idt_reg_clear_bits() functions
- Add appropriate naming of function idt_get_mw_size()
- Fix some documentation notes
- Replace symbolic permission S_IRUSR with octal 0400

Changelog v4:
- Return ~0 on read from registers with invalid address
- Don't check bits validity on registers bits clearing
- Keep up driver loading (just print a warning) if there is no any peer
NTBs found
- Fix unnecessary branching logic
- Fix some documentation notes

Changelog v5:
- Fix minor documentation issues
- Replace writel/readl with iowrite32/ioread32 methods
- Discard dev_*() wrappers with origins
- Use pci_alloc_irq_vectors() for IRQ number and ISR initialization
- Use Mananged Device Resource as much as possible:
devm_request_threaded_irq(), pcim_iomap_regions_request_all()

 drivers/ntb/hw/Kconfig  |1 +
 drivers/ntb/hw/Makefile |1 +
 drivers/ntb/hw/idt/Kconfig  |   31 +
 drivers/ntb/hw/idt/Makefile |1 +
 drivers/ntb/hw/idt/ntb_hw_idt.c | 2600 +++
 drivers/ntb/hw/idt/ntb_hw_idt.h | 1149 +
 6 files changed, 3783 insertions(+)
 create mode 100644 drivers/ntb/hw/idt/Kconfig
 create mode 100644 drivers/ntb/hw/idt/Makefile
 create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.c
 create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.h

diff --git a/drivers/ntb/hw/Kconfig b/drivers/ntb/hw/Kconfig
index 7116472..a89243c 100644
--- a/drivers/ntb/hw/Kconfig
+++ b/drivers/ntb/hw/Kconfig
@@ -1,2 +1,3 @@
 source "drivers/ntb/hw/amd/Kconfig"
+source "drivers/ntb/hw/idt/Kconfig"
 source "drivers/ntb/hw/intel/Kconfig"
diff --git a/drivers/ntb/hw/Makefile b/drivers/ntb/hw/Makefile
index 532e085..87332c3 100644
--- a/drivers/ntb/hw/Makefile
+++ b/drivers/ntb/hw/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_NTB_AMD)  += amd/
+obj-$(CONFIG_NTB_IDT)  += idt/
 obj-$(CONFIG_NTB_INTEL)+= intel/
diff --git a/drivers/ntb/hw/idt/Kconfig b/drivers/ntb/hw/idt/Kconfig
new file mode 100644
index 000..b360e56
--- /dev/null
+++ b/drivers/ntb/hw/idt/Kconfig
@@ -0,0 +1,31 @@
+config NTB_IDT
+   tristate "IDT PCIe-switch Non-Transparent Bridge support"
+   depends on PCI
+   help
+This driver supports NTB of cappable IDT PCIe-switches.
+
+Some of the pre-initializations must be made before IDT PCIe-switch
+exposes it NT-functions correctly. It should be done by either proper
+initialisation of EEPROM connected to master smbus of the switch or
+by BIOS using slave-SMBus interface changing corresponding registers
+value. Evidently it must be done before PCI bus enumeration is
+finished in Linux kernel.
+
+First of all partitions must be activated and properly assigned to all
+the ports with NT-functions intended to be activated (see SWPARTxCTL
+and SWPORTxCTL registers). Then all NT-function BARs must be enabled
+with chosen valid aperture. For memory windows related BARs the
+aperture settings shall determine the maximum size of memory windows
+accepted by a BAR. Note that BAR0 must map PCI configuration space
+registers.
+
+It's worth to note, that since a part of this driver relies on the
+BAR settings of peer NT-functions, the BAR setups can't be done over
+

[PATCH v5] NTB: Add IDT 89HPESxNTx PCIe-switches support

[Serge Semin]

IDT 89HPESxNTx device series is PCIe-switches, which support
Non-Transparent bridging between domains connected to the device ports.
Since new NTB API exposes multi-port interface and messaging API, the
IDT NT-functions can be now supported in the kernel. This driver adds
the following functionality:
1) Multi-port NTB API to have information of possible NT-functions
activated in compliance with available device ports.
2) Memory windows of direct and look up table based address translation
with all possible combinations of BARs setup.
3) Traditional doorbell NTB API.
4) One-on-one messaging NTB API.

There are some IDT PCIe-switch setups, which must be done before any of
the NTB peers started. It can be performed either by system BIOS via
IDT SMBus-slave interface or by pre-initialized IDT PCIe-switch EEPROM:
1) NT-functions of corresponding ports must be activated using
SWPARTxCTL and SWPORTxCTL registers.
2) BAR0 must be configured to expose NT-function configuration
registers map.
3) The rest of the BARs must have at least one memory window
configured, otherwise the driver will just return an error.
Temperature sensor of IDT PCIe-switches can be also optionally
activated by BIOS or EEPROM.
(See IDT documentations for details of how the pre-initialization can
be done)

Signed-off-by: Serge Semin 
Acked-by: Allen Hubbe 

---

Changelog v2:
- Fix minor checkpatch.pl issues
- Get rid of obfuscating macros

Changelog v3:
- No write to registers if address is either out of bound or unaligned
- Fix idt_reg_set_bits()/idt_reg_clear_bits() methods race condition
- Fix invalid argument of write method called from
idt_reg_set_bits()/idt_reg_clear_bits() functions
- Add appropriate naming of function idt_get_mw_size()
- Fix some documentation notes
- Replace symbolic permission S_IRUSR with octal 0400

Changelog v4:
- Return ~0 on read from registers with invalid address
- Don't check bits validity on registers bits clearing
- Keep up driver loading (just print a warning) if there is no any peer
NTBs found
- Fix unnecessary branching logic
- Fix some documentation notes

Changelog v5:
- Fix minor documentation issues
- Replace writel/readl with iowrite32/ioread32 methods
- Discard dev_*() wrappers with origins
- Use pci_alloc_irq_vectors() for IRQ number and ISR initialization
- Use Mananged Device Resource as much as possible:
devm_request_threaded_irq(), pcim_iomap_regions_request_all()

 drivers/ntb/hw/Kconfig  |1 +
 drivers/ntb/hw/Makefile |1 +
 drivers/ntb/hw/idt/Kconfig  |   31 +
 drivers/ntb/hw/idt/Makefile |1 +
 drivers/ntb/hw/idt/ntb_hw_idt.c | 2600 +++
 drivers/ntb/hw/idt/ntb_hw_idt.h | 1149 +
 6 files changed, 3783 insertions(+)
 create mode 100644 drivers/ntb/hw/idt/Kconfig
 create mode 100644 drivers/ntb/hw/idt/Makefile
 create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.c
 create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.h

diff --git a/drivers/ntb/hw/Kconfig b/drivers/ntb/hw/Kconfig
index 7116472..a89243c 100644
--- a/drivers/ntb/hw/Kconfig
+++ b/drivers/ntb/hw/Kconfig
@@ -1,2 +1,3 @@
 source "drivers/ntb/hw/amd/Kconfig"
+source "drivers/ntb/hw/idt/Kconfig"
 source "drivers/ntb/hw/intel/Kconfig"
diff --git a/drivers/ntb/hw/Makefile b/drivers/ntb/hw/Makefile
index 532e085..87332c3 100644
--- a/drivers/ntb/hw/Makefile
+++ b/drivers/ntb/hw/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_NTB_AMD)  += amd/
+obj-$(CONFIG_NTB_IDT)  += idt/
 obj-$(CONFIG_NTB_INTEL)+= intel/
diff --git a/drivers/ntb/hw/idt/Kconfig b/drivers/ntb/hw/idt/Kconfig
new file mode 100644
index 000..b360e56
--- /dev/null
+++ b/drivers/ntb/hw/idt/Kconfig
@@ -0,0 +1,31 @@
+config NTB_IDT
+   tristate "IDT PCIe-switch Non-Transparent Bridge support"
+   depends on PCI
+   help
+This driver supports NTB of cappable IDT PCIe-switches.
+
+Some of the pre-initializations must be made before IDT PCIe-switch
+exposes it NT-functions correctly. It should be done by either proper
+initialisation of EEPROM connected to master smbus of the switch or
+by BIOS using slave-SMBus interface changing corresponding registers
+value. Evidently it must be done before PCI bus enumeration is
+finished in Linux kernel.
+
+First of all partitions must be activated and properly assigned to all
+the ports with NT-functions intended to be activated (see SWPARTxCTL
+and SWPORTxCTL registers). Then all NT-function BARs must be enabled
+with chosen valid aperture. For memory windows related BARs the
+aperture settings shall determine the maximum size of memory windows
+accepted by a BAR. Note that BAR0 must map PCI configuration space
+registers.
+
+It's worth to note, that since a part of this driver relies on the
+BAR settings of peer NT-functions, the BAR setups can't be done over
+kernel PCI fixups. That's why the alternative