Currently developed NTB-bus supports AMD and Intel Non-transparent PCIe-bridges only. These bridges implement "synchronous" interfaces, which means that memory window translated addresses can be direcly specified from one side to another by writing to a corresponding value to a register. Unlike Intel and AMD there is another vendor - IDT, which produces synchronous and asynchrnous NTB PCIe-switches. If synchronous NTB IDT PCIe-switches works almost the same way as AMD and Intel devices do, the asynchronous ones don't. Due to the registers field specification the translated base addresses of memory windows can not be directly specified to a peer. To solve the problem IDT asynchronous PCIe-switches implement messaging interface to send any kind of information from one RC to another including translated base addresses. In the framework of this patches we developed a set of callback functions for the NTB-bus devices programming interface and drivers for the NT-functions of the following IDT PCIe-switches: - 89HPES24NT6AG2, - 89HPES32NT8AG2, - 89HPES32NT8BG2, - 89HPES12NT12G2, - 89HPES16NT16G2, - 89HPES24NT24G2, - 89HPES32NT24AG2, - 89HPES32NT24BG2, and a set of NTB client drivers to test each subsystem of the IDT NTB drivers separately.
The patches source code was tested on the hardware with IDT 89HPES32NT8AG2 PCIe-switch connected to root complexes running x64 big endian and x32 little endian CPUs. The code was warningless built by x86_64, MIPS64 BE and MIPS32 LE GNU GCC compilers. To make it easely applicable to the Linux kernel NTB subsystem, the patches were taken from the top of the "ntb-next" branch of the forked NTB git repository: https://github.com/fancer/ntb/tree/ntb-next (it is the fork of https://github.com/jonmason/ntb/tree/ntb-next) so the hash of the last preceding branch commit was: 04a819bbee3fa4dfe22034e2724dab4efb63f0bd There are a some types of checkpatch warnings I left unfixed in the framework of the source code. Here is the reason why: 1) WARNING: Comparisons should place the constant on the right side of the test Reason: The whole code is written for typos safety. Writing the constants on the left side of comparisons prevents the code from compiling if "=" and "==" are mistakenly confused. 2) WARNING: braces {} are not necessary for any arm of this statement Reason: Even though the braces aren't really necessary, I left them in there to make the code better distinguishable from the rest of the code. 3) WARNING: Avoid crashing the kernel - try using WARN_ON & recovery code rather than BUG() or BUG_ON() Reason: If the BUG()/BUG_ON() macro were invoked, something was really wrong, so the developer should make sure all the configurations are correct. Otherwise one should not use the driver. 4) WARNING: line over 80 characters Reason: Most of these warnings appear due to just 3-5 excess characters, which can not be excluded without reducing the code readability. The exception is the registers tables, which are described in the item 8). 5) WARNING: Block comments use a trailing */ on a separate line Reason: All of those warnings mean that the comment closure pattern"*/" is specified in the end of the last comment line. If one puts the pattern to a next line, comments visually get separated from the code they are made for. It reduces the readability. 6) WARNING: quoted string split across lines Reason: There was no way to reduce string lengths, so the only way to split it across lines. 7) WARNING: do not add new typedefs Reason: They are really needed there and added with full respect to the coding style document. 8) ERROR: Macros with multiple statements should be enclosed in a do - while loop ERROR: Macros with complex values should be enclosed in parentheses Reason: Those macros are used to create a very suitable registers table using "enum-switch-case-macro" design pattern with o(1) access to the register address, size and description. So adding the parentheses and do - while statements will break the pattern. Additionally they aren't used anywhere else, than in the framework of the pattern functions. Of course all of these warnings are discussable, except the last one. It would be really painful to remove it from the code =) As one can see I thoroughly annotated and commented the drivers source code, so it would be easier to study it by kernel hackers. Here is what the current design of NTB-bus and IDT NTB function drivers lacks of: 1) No support of Punch-Through functionlity of IDT PCIe-switches 2) DMA is not supported by the current revision of IDT NTB driver 3) Theoretically local and peer root complexes can have different CPU architecture like x86 and x64. In this way, the dma_addr_t type will have different size, which may lead to errors when x64 RC allocates a shared buffer above 4GB memory. One more suggestion. Since after adding this set of patches there will be two types of devices the NTB-bus supports, we may want to implement the traditional linux kernel bus device-drivers matching algorithm using some new struct ntb_device_id and id_tables. Thanks, ============================= Serge V. Semin Leading Programmer Embedded SW development group T-platforms ============================= Signed-off-by: Serge Semin <fancer.lan...@gmail.com> fancer (3): ntb: Add asynchronous devices support to NTB-bus interface ntb: IDT 89HPES*NT* PCIe-switches NTB device driver ntb: Test client drivers for asynchronous NTB devices drivers/ntb/Kconfig | 4 +- drivers/ntb/hw/Kconfig | 1 + drivers/ntb/hw/Makefile | 6 +- drivers/ntb/hw/amd/ntb_hw_amd.c | 49 +- drivers/ntb/hw/idt/Kconfig | 21 + drivers/ntb/hw/idt/Makefile | 5 + drivers/ntb/hw/idt/ntb_hw_idt.c | 4050 ++++++++++++++++++++++++++++++++ drivers/ntb/hw/idt/ntb_hw_idt.h | 390 +++ drivers/ntb/hw/idt/ntb_hw_idt_quirks.c | 163 ++ drivers/ntb/hw/idt/ntb_hw_idt_quirks.h | 114 + drivers/ntb/hw/idt/ntb_hw_idt_regmap.h | 877 +++++++ drivers/ntb/hw/intel/ntb_hw_intel.c | 59 +- drivers/ntb/ntb.c | 86 +- drivers/ntb/ntb_transport.c | 19 +- drivers/ntb/test/Kconfig | 32 + drivers/ntb/test/Makefile | 9 +- drivers/ntb/test/ntb_db_test.c | 677 ++++++ drivers/ntb/test/ntb_msg_test.c | 736 ++++++ drivers/ntb/test/ntb_mw_test.c | 1531 ++++++++++++ drivers/ntb/test/ntb_perf.c | 16 +- drivers/ntb/test/ntb_pingpong.c | 5 + drivers/ntb/test/ntb_tool.c | 25 +- include/linux/ntb.h | 600 ++++- 23 files changed, 9309 insertions(+), 166 deletions(-) create mode 100644 drivers/ntb/hw/idt/Kconfig create mode 100644 drivers/ntb/hw/idt/Makefile create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.c create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt.h create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt_quirks.c create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt_quirks.h create mode 100644 drivers/ntb/hw/idt/ntb_hw_idt_regmap.h create mode 100644 drivers/ntb/test/ntb_db_test.c create mode 100644 drivers/ntb/test/ntb_msg_test.c create mode 100644 drivers/ntb/test/ntb_mw_test.c -- 2.6.6