RE: [PATCHv1] RDMA/ocrdma: Fixed CONFIG_VLAN_8021Q.
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Thursday, August 16, 2012 9:41 AM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org
> Subject: Re: [PATCHv1] RDMA/ocrdma: Fixed CONFIG_VLAN_8021Q.
>
> On Sat, Aug 11, 2012 at 6:28 AM, Parav Pandit
> wrote:
> > +static struct net_device *ocrdma_get_real_netdev(struct net_device
> > +*netdev) { #if IS_ENABLED(CONFIG_VLAN_8021Q)
> > + return vlan_dev_real_dev(netdev); #else
> > + return netdev;
> > +#endif
> > +}
>
> As I said before, I don't think this wrapper is needed, and even if it were,
> it
> would be much better to write it without using the preprocessor (as I said,
> you can do "if (IS_ENABLED(...))" in C code now too).
>
> I'm going to stick to my simpler version unless there is something wrong with
> it...
[PP] O.k. I am fine with your solution too.
>
> - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: BUG: RDMA/ocrdma calls invalid vlan_dev_real_dev()
I'll provide you fix in short while.
Parav
> -Original Message-
> From: Fengguang Wu [mailto:fengguang...@intel.com]
> Sent: Friday, August 10, 2012 5:39 AM
> To: Roland Dreier
> Cc: linux-rdma@vger.kernel.org; Pandit, Parav; Sean Hefty; linux-
> ker...@vger.kernel.org
> Subject: Re: BUG: RDMA/ocrdma calls invalid vlan_dev_real_dev()
>
> On Thu, Aug 09, 2012 at 04:54:37PM -0700, Roland Dreier wrote:
> > thanks for the report. I assume the system doesn't actually have ocrdma
> hw?
>
> Yeah, it's a test boot inside KVM.
>
> Thanks,
> Fengguang
>
> > - R.
> > On Aug 9, 2012 3:00 AM, "Fengguang Wu"
> wrote:
> >
> > > Hi Parav,
> > >
> > > commit fe2caefcdf ("RDMA/ocrdma: Add driver for Emulex OneConnect
> > > IBoE RDMA adapter") triggers the below kernel BUG for the attached
> config.
> > >
> > > [ 280.861196] kernel BUG at
> > > /c/kernel-tests/src/stable/include/linux/if_vlan.h:113!
> > > [ 280.861196] invalid opcode: [#1] PREEMPT [ 280.861196] CPU
> > > 0 [ 280.861196] Pid: 304, comm: ip Not tainted 3.6.0-rc1 #1 Bochs
> > > Bochs [ 280.861196] RIP: 0010:[]
> > > []
> > > ocrdma_inet6addr_event+0x4/0x6
> > > [ 280.861196] RSP: 0018:8800066a1548 EFLAGS: 0202 [
> > > 280.861196] RAX: 0001 RBX: RCX:
> > >
> > > [ 280.861196] RDX: 880006b6b400 RSI: 0001 RDI:
> > > 8207ecc0
> > > [ 280.861196] RBP: 8800066a1548 R08: R09:
> > > 8109b657
> > > [ 280.861196] R10: R11: 81e0f318 R12:
> > >
> > > [ 280.861196] R13: 8207ecc0 R14: R15:
> > >
> > > [ 280.861196] FS: 7f025c12f700() GS:81dfb000()
> > > knlGS:
> > > [ 280.861196] CS: 0010 DS: ES: CR0: 8005003b [
> > > 280.861196] CR2: 7fe4e1d3 CR3: 06b65000 CR4:
> > > 06b0
> > > [ 280.861196] DR0: DR1: DR2:
> > >
> > > [ 280.861196] DR3: DR6: DR7:
> > >
> > > [ 280.861196] Process ip (pid: 304, threadinfo 8800066a,
> > > task
> > > 880006a7c2c0)
> > > [ 280.861196] Stack:
> > > [ 280.861196] 8800066a1598 8109b5a2 880006b6b400
> > > 0001
> > > [ 280.861196] 8800066a1598 0001 880006b6b400
> > > [ 280.861196] 820ad9b0
> > > 8800066a15f8
> > > 8109b6b7
> > > [ 280.861196] Call Trace:
> > > [ 280.861196] [] notifier_call_chain+0x60/0x90 [
> > > 280.861196] []
> > > __atomic_notifier_call_chain+0x60/0x92
> > > [ 280.861196] [] ?
> > > atomic_notifier_chain_unregister+0x46/0x46
> > > [ 280.861196] []
> > > atomic_notifier_call_chain+0xf/0x11
> > > [ 280.861196] [] ipv6_add_addr+0x333/0x388 [
> > > 280.861196] [] ? ipv6_add_addr+0x55/0x388 [
> > > 280.861196] [] add_addr+0x12/0x5c [ 280.861196]
> > > [] init_loopback+0x7b/0x7f [ 280.861196]
> > > [] addrconf_notify+0x178/0x2d4 [ 280.861196]
> > > [] notifier_call_chain+0x60/0x90 [ 280.861196]
> > > [] __raw_notifier_call_chain+0x9/0xb [
> > > 280.861196] [] raw_notifier_call_chain+0xf/0x11 [
> > > 280.861196] [] call_netdevice_notifiers+0x45/0x4a
> > > [ 280.861196] [] __dev_notify_flags+0x32/0x56 [
> > > 280.861196] [] dev_change_flags+0x43/0x4e [
> > > 280.861196] [] do_setlink+0x2da/0x7f6 [
> > > 280.861196] [] ? native_sched_clock+0x38/0x68 [
> > > 280.861196] [] ? sched_clock+0x9/0xd [
> > > 280.861196] [] ?
> > > sched_clock_local.constprop.2+0xd/0x78
> > > [ 280.861196] [] ? sched_clock_cpu+0x7b/0x89 [
> > > 280.861196] [] rtnl_newlink+0x264/0x438 [
> > > 280.861196] [] ? rtnl_newlink+0xba/0x438 [
> > > 280.861196] [] ? avc_has_perm_noaudit+0xd1/0xe3 [
> > > 280.861196] [] ? avc_has_perm_noaudit+0x22/0xe3 [
> > > 280.861196] [] rtnetlink_rcv_msg+0x22c/0x23b [
> > > 280.861196] [] ? rtnl_lock+0x12/0x14 [
> > > 280.861196] [] ? __rtnl_unlock+0x12/0x12 [
> > > 280.861196] [] netlink_rcv_skb+0x3d/0x8a [
> > > 280.861196] [] rtnetlink_rcv+0x21/0x28 [
> > > 280.861196] [] netlink_unicast+0x12c/0x1b8 [
> > > 280.861196] [] netlink_sendmsg+0x212/0x29a [
> > > 280.861196] [] sock_sendmsg+0x9e/0xbf [
> > > 280.861196] [] __sys_sendmsg+0x248/0x2d5 [
> > > 280.861196] [] ? _raw_spin_unlock_irq+0x34/0x50 [
> > > 280.861196] [] ?
> > > finish_task_switch.constprop.48+0x72/0xd9
> > > [ 280.861196] [] ?
> > > finish_task_switch.constprop.48+0x34/0xd9
> > > [ 280.861196] [] ? __schedule+0x501/0x607 [
> > > 280.861196] [] ? put_lock_stats.isra.17+0xe/0x28
> > > [ 280.861196] [] ?
> > > lock_release_holdtime+0xcd/0xd5 [ 280.861196] []
> > > sys_sendmsg+0x3d/0x5e [ 280.861196] []
> > > system_call_fastpath+0x16/0x1b [ 280.861196] Code: 00 00 ad de 48
> > > 89 93 a8 0b 00 00 e8 c9 2f 2e 00 48 8d bb b0 0b 00 00 48 c7 c6 86 f0
> > > 6d 81 e8 b0 b2 9e ff 59 5b 5d c3 55 48
RE: [Q] How to tranfer a file which is over 2GB(2^31) size in RDMA network?
> -Original Message- > From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma- > ow...@vger.kernel.org] On Behalf Of Hiroyuki Sato > Sent: Monday, July 02, 2012 7:20 PM > To: linux-rdma > Subject: [Q] How to tranfer a file which is over 2GB(2^31) size in RDMA > network? > > Dear developers. > > I'm writing simple file transfer program. > > I would like to know about the following. > Q: How to tranfer a file which is over 2GB(2^31) size in RDMA network? > > Please imagine to transfer whole DVD(4.7GB) file via RDMA networok. > > The maximum message size of RC is 2^31. > so I can't transfer it with one RDMA message. > > Maybe It must split multiple parts. > When I sent first part with the following sequence, how to transfer second > part? > > Do I have to call qp ibv_destroy_qp and recreate new qp? > Or can I reuse current qp? > You can use the same QP to send multiple messages of same or different size. Receive side needs to have sufficient memory to receive 2GB of data, which you should have posted using post_recv(). Since you are going to divide a file/consumer application data to multiple RDMA messages, make sure sender and receiver agrees to a message size of interest. This is the simplest way to do via RDMA_SEND and RECV buffers. You can further do it via sharing the stag before invoking post_send/recv and use RDMA Write or READ operations and don't need to synchronize the message size. Sender can send/split data into one or more WRITE messages followed by ending it with write_immidiate or send_immidiate to notify the peer of file transfer done. In a third way, you can void writing the application and use SDP protocol to send your file via FTP/SFTP, in which FTP server and client application sockets use the SDP sockets instead of TCP sockets. > I'm looking for similar example but I can't find it. > Any information is welcome. > > Thank you for your advice. > > Sincerely > > -- > Hiroyuki Sato. > > > File transfer sequence (1st part) > > ibv_open_device > ibv_alloc_pd > ibv_reg_mr > ibv_create_cq > ibv_create_qp > ibv_modify_qp(RESET->INIT) > ibv_post_recv > exchange lid, sid,qpn > ibv_connect > ibv_modify_qp(INIT->RTR) > ibv_modify_qp(RTR->RTS) > ibv_post_send > ... > > > > Environment > OS Scientific Linux 6.2 > OFED: 1.5.4.1 > > -- > Hiroyuki Sato > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the > body of a message to majord...@vger.kernel.org More majordomo info at > http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
fmr pool and remap doubt
Hi, I am trying to understand remapping functionality and fmr_pool.c. Looking back at old thread: http://lists.openfabrics.org/pipermail/general/2006-February/017198.html Can you please confirm whether my understanding is correct or not. 1. max_map_per_fmr indicates that - different memory pages can be remapped (again) without invoking unmap_fmr(). Basically previous mapping is over-written on every map_phys_fmr() call with new mapping without doing unmapping. 2. adapter indicates above limit - how many times old mapping can be overwritten (remapped) before invoking unmap_fmr(). 3. Remapping allows faster operation compare to map(), unmap() sequence, due to which unmap_fmr() is mostly done in worker threads in SDP, RDS etc consumers. Is above understanding correct? Regards, Parav Pandit -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
be2net: when can I expect roce support patch will be merged?
Hi, Did you get chance to merge below be2net patch for supporing RoCE driver? http://marc.info/?l=linux-rdma&m=133279326217836&w=2 Once this is done, Roland can merge ocrdma RoCE driver addition to his tree. This NIC driver patch is required to merge ocrdma patch. When can I expect this patch to be merged to net-next git tree? Let me know if there any issue with the patch that I got to resolve. Regards, Parav Pandit -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
Got it. You did mention about typedef in email chain, but I understood as
different way to achieve same.
I reviewed my code and found that most of the fields between driver-adapter
doesn't need attribute.
So far (a) removing packed and (b) BUILD_BUG_ON looks sufficient for current
set of structures. Initial test suffice the need.
Thanks.
Parav
> -Original Message-
> From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com]
> Sent: Friday, March 23, 2012 10:24 PM
> To: Pandit, Parav
> Cc: david.lai...@aculab.com; rol...@purestorage.com; linux-
> r...@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> On Fri, Mar 23, 2012 at 07:03:37AM -0700, parav.pan...@emulex.com wrote:
>
> > > David is saying you will get a 12 byte struct and fieldb will be
> > > unaligned. Since
> > > 12 is aligned to 4 no padding is added.
> >
> > So I decided to experiment above example before implementing in
> > driver. However I find structure of 16 bytes (instead of 12) with
> > padding after fielda in below example. Am I missing some compiler
> > option or syntax error in attribute? Sorry to ask this silly question.
> > I tried __attribute__((__aligned__(4))); too based on usage in other
> > kernel code.
>
> I got the syntax wrong for that specific case (it is a little unintuitive..
> IMHO,
> capping the alignment of a container should cap the alignment of all
> members, otherwise it is nonsense!):
>
> typedef uint64_t u64_unaligned_8 __attribute__((__aligned__(4)));
>
> struct foo {
> uint32_t fielda;
> u64_unaligned_8 fieldb;
> };
>
> struct foo2 {
> uint32_t fielda;
> uint64_t fieldb;
> };
>
> int main(int argc,const char *argv[])
> {
> printf("sizeof(foo) = %zu, fieldb = %zu\n",sizeof(struct foo),
> offsetof(struct foo,fieldb));
> printf("sizeof(foo2) = %zu, fieldb = %zu\n",sizeof(struct foo2),
> offsetof(struct foo2,fieldb));
> return 0;
> }
>
> sizeof(foo) = 12, fieldb = 4
> sizeof(foo2) = 16, fieldb = 8
>
> gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3)
>
> Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com]
> Sent: Friday, March 23, 2012 4:14 AM
> To: Pandit, Parav
> Cc: david.lai...@aculab.com; rol...@purestorage.com; linux-
> r...@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> On Thu, Mar 22, 2012 at 02:20:28PM -0700, parav.pan...@emulex.com
> wrote:
> > I got a question here lately.
> >
> > aligned directive will ensure that it will fall on boundary. Say
> > aligned(4) ensures that structure is aligned to 4 byte boundary.
> > Compiler can (at least theoretically) still have 4 byte structure
> > aligned to 8 byte boundary on 64-bit platform (which is 4 byte aligned
> > too).
>
> There are very specific rules defined in the platform's ABI for how C
> structures are layed out in memory, each ABI (ie CPU) has its own specific
> quirks, but broadly in Linux land you can boil it down to:
>
> 1) The alignment of a structure is the greatest alignment of all the
>members
> 2) Each member is aligned to its alignment.
>
> The alignment of the structure drives the total size of the structure, and
> specifically the padding added at the end to reach that alignment.
>
> So, no, a compiler that increased the alignment of a struct with one
> u32 to 8 would violate the various ABIs and not be usuable for Linux. It is
> important to bear in mind that Linux targets a particular set of ABI
> conventions, and it is not 'anything goes'.
>
> > struct {
> > u32 field;
> > };
>
> So in this case: the u32 is aligned to 4, the structure is aligned to
> 4 and the total size of the structure is 4 on everything linux supports.
>
> > struct {
> > u64 fielda
> > u32 field;
> > };
>
> In this case: On 64 bit: the u64 is aligned to 8 and the u32 is aligned to 4.
> So
> the structure is aligned to 8. A pad is inserted at the end of the struct to
> bring
> it out. On 32 bit, the u64 is aligned to 4, so the struct is aligned to 4, so
> no pad
> is added.
>
> > struct {
> > __float128 fielda
> > u32 field;
> > };
>
> In this case the float128 is aligned to 16 and thus the structure is aligned
> to 16
> and 12 pad bytes are added.
>
> > However requirement is to have this structure only 4 byte size(
> > because adapter excepts it to be 4B sise) and therefor packed is used.
> > I don't know the way to ensure size of 4 byte and alignment too. Or I
> > am misunderstanding?
>
> Yes, you are mis-understanding the rules for padding.. Structures are only
> padded out to their alignment, which depends on their constituent types.
> This is so arrays of structures have each array element starting on its
> natural
> alignment.
>
> The aligned attribute overrides the automatic determination of the
> alignment based on the contents and just forces it.
>
> So, as an example, if you have this hardware layout:
>
> struct {
> u32 fielda;
> u64 fieldb;
> } attribute ((aligned(4));
>
> David is saying you will get a 12 byte struct and fieldb will be unaligned.
> Since
> 12 is aligned to 4 no padding is added.
So I decided to experiment above example before implementing in driver. However
I find structure of 16 bytes (instead of 12) with padding after fielda in below
example.
Am I missing some compiler option or syntax error in attribute? Sorry to ask
this silly question.
I tried __attribute__((__aligned__(4))); too based on usage in other kernel
code.
#include
struct foo {
u32 a;
u64 b;
} __attribute__((aligned(4)));
static int __init main_init(void)
{
printk("<1> sizeof foo=%ld\n", sizeof(struct foo));
printk("<1> offset of a=%ld\n", offsetof(struct foo, a));
printk("<1> offset of b=%ld\n", offsetof(struct foo, b));
return 0;
}
static void __exit main_exit(void)
{
}
module_init(main_init);
module_exit(main_exit);
Output:
Mar 24 05:44:10 parav-sles11-sp1-64 kernel: [2006356.094931] sizeof foo=16
Mar 24 05:44:10 parav-sles11-sp1-64 kernel: [2006356.094934] offset of a=0
Mar 24 05:44:10 parav-sles11-sp1-64 kernel: [2006356.094936] offset of b=8
Gcc version is below.
Using built-in specs.
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info
--mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64
--enable-languages=c,c++,objc,fortran,obj-c++,java,ada
--enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.3
--enable-ssp --disable-libssp --with-bugurl=http://bugs.opensuse.org/
--with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap
--with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit
--enable-libstdcxx-allocator=new --disable-libstdcxx-pch
--enable-version-specific-runtime-libs --program-suffix=-4.3
--enable-linux-futex --without-system-libunwind --with-cpu=generic
--build=x86_64-suse-linux
Thread model: posix
gcc version 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)
> For hard
RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
I got a question here lately.
aligned directive will ensure that it will fall on boundary.
Say aligned(4) ensures that structure is aligned to 4 byte boundary.
Compiler can (at least theoretically) still have 4 byte structure aligned to 8
byte boundary on 64-bit platform (which is 4 byte aligned too).
struct {
u32 field;
} attribute ((aligned(4));
However requirement is to have this structure only 4 byte size( because adapter
excepts it to be 4B sise) and therefor packed is used.
I don't know the way to ensure size of 4 byte and alignment too.
Or I am misunderstanding?
Parav
> -Original Message-
> From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-
> ow...@vger.kernel.org] On Behalf Of parav.pan...@emulex.com
> Sent: Friday, March 23, 2012 2:41 AM
> To: jguntho...@obsidianresearch.com
> Cc: david.lai...@aculab.com; rol...@purestorage.com; linux-
> r...@vger.kernel.org; net...@vger.kernel.org
> Subject: RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
>
>
> > -Original Message-
> > From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com]
> > Sent: Friday, March 23, 2012 2:28 AM
> > To: Pandit, Parav
> > Cc: david.lai...@aculab.com; rol...@purestorage.com; linux-
> > r...@vger.kernel.org; net...@vger.kernel.org
> > Subject: Re: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA
> > adapter
> >
> > On Thu, Mar 22, 2012 at 01:52:30PM -0700, parav.pan...@emulex.com
> > wrote:
> >
> > > > This can be used to force 32bit alignment in amd64 code in order
> > > > to match definitions in 32bit userspace.
> > > > For new things it would make sense to force 64bit alignment of
> > > > 64bit fields for 32bit code.
> > >
> > > o.k. so I'll use aligned attribute to align user-kernel interface
> > > data structure to 8 byte boundary. That should work for 32-bit and
> > > 64-bit user and kernel space and does't hurt performance either?
> >
> > If the structure is only for user/kernel interfacing then it is much
> > better to add explicit padding fields to naturally place 64 bit
> > quantities on an 8 byte alignment than to mess with gcc specific
> > attributes (user space has a much wide choice of compilers).
> >
> > This was David's second suggestion. Better to do this now before the
> > driver is accepted :)
> >
> o.k. I'll align them to naturally 8 byte boundary.
>
> > Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the
> body of a message to majord...@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message- > From: Jason Gunthorpe [mailto:jguntho...@obsidianresearch.com] > Sent: Friday, March 23, 2012 2:28 AM > To: Pandit, Parav > Cc: david.lai...@aculab.com; rol...@purestorage.com; linux- > r...@vger.kernel.org; net...@vger.kernel.org > Subject: Re: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA > adapter > > On Thu, Mar 22, 2012 at 01:52:30PM -0700, parav.pan...@emulex.com > wrote: > > > > This can be used to force 32bit alignment in amd64 code in order to > > > match definitions in 32bit userspace. > > > For new things it would make sense to force 64bit alignment of 64bit > > > fields for 32bit code. > > > > o.k. so I'll use aligned attribute to align user-kernel interface data > > structure to 8 byte boundary. That should work for 32-bit and 64-bit > > user and kernel space and does't hurt performance either? > > If the structure is only for user/kernel interfacing then it is much better to > add explicit padding fields to naturally place 64 bit quantities on an 8 byte > alignment than to mess with gcc specific attributes (user space has a much > wide choice of compilers). > > This was David's second suggestion. Better to do this now before the driver is > accepted :) > o.k. I'll align them to naturally 8 byte boundary. > Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: David Laight [mailto:david.lai...@aculab.com]
> Sent: Wednesday, March 21, 2012 10:02 PM
> To: Roland Dreier; Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
>
> > > - Header file for userspace library and kernel driver interface.
> >
> > > +struct ocrdma_alloc_ucontext_resp {
> > > + u32 dev_id;
> > > + u32 wqe_size;
> > > + u32 max_inline_data;
> > > + u32 dpp_wqe_size;
> > > + u64 ah_tbl_page;
> > > + u32 ah_tbl_len;
> > > + u32 rsvd;
> > > + u8 fw_ver[32];
> > > + u32 rqe_size;
> > > + u64 rsvd1;
> > > +} __packed;
> >
> > If I'm reading this correctly, you have the 8-byte rsvd1 member at an
> > offset only aligned to 4 bytes, because of the __packed directive. It
> > would be much better to have these structures laid out so they are
> > naturally the same on both 32-bit and 64-bit ABIs, and get rid of the
> > __packed directive, which wrecks gcc code generation in some cases.
> >
>
> gcc also supports defining types that have non-standard alignment
> constraints that can be used to force the same alignment for 64bit fields
> between i386 and amd64.
> Probably __attribute__((aligned,n)) or similar.
>
> This can be used to force 32bit alignment in amd64 code in order to match
> definitions in 32bit userspace.
> For new things it would make sense to force 64bit alignment of 64bit fields
> for 32bit code.
o.k. so I'll use aligned attribute to align user-kernel interface data
structure to 8 byte boundary.
That should work for 32-bit and 64-bit user and kernel space and does't hurt
performance either?
Driver-adapter structures will be aligned to 4 byte boundary using aligned
attribute instead of packed.
>
> Adding __packed (rather than 32bit alignment) forces the compiler to
> generate byte by byte accesses for all the fields on systems that can't do
> misaligned accesses in hardware (eg sparc).
>
> David
>
>
> David
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message- > From: Roland Dreier [mailto:rol...@purestorage.com] > Sent: Thursday, March 22, 2012 9:30 PM > To: Pandit, Parav > Cc: sean.he...@intel.com; linux-rdma@vger.kernel.org > Subject: Re: [PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA > adapter > > On Thu, Mar 22, 2012 at 7:27 AM, wrote: > > I prefer to have it in this first patch and once the internal test cycle is > > finish > (with removed code) in few days, will submit the separate patch? > > It's fine to leave some of these items for future cleanups post-merge. > Just please try to get to the cleanups someday unlike some other drivers... Sure. Item is list and tracking the same. Mostly likely it's going to be 2nd patch after this first release before adding any new features. > (*cough* nes *cough*) > > - R. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Hefty, Sean [mailto:sean.he...@intel.com]
> Sent: Wednesday, March 21, 2012 10:49 PM
> To: Pandit, Parav; linux-rdma@vger.kernel.org
> Subject: RE: [PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +struct ocrdma_cq {
> > + struct ib_cq ibcq;
> > + struct ocrdma_dev *dev;
>
> nit: There are several structures where you store ocrdma_dev *. You can
> remove these and use the struct ib_* to reach it as well.
Removed from all the ocrdma_xxx structures except ocrdma_eq and ocrdma_qp.
ocrdma_eq is anyway necessary.
Removing it from ocrdma_qp will require a lot bigger test cycle at my end.
I prefer to have it in this first patch and once the internal test cycle is
finish (with removed code) in few days, will submit the separate patch?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 5/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Hefty, Sean [mailto:sean.he...@intel.com]
> Sent: Thursday, March 22, 2012 6:14 AM
> To: Pandit, Parav; linux-rdma@vger.kernel.org
> Subject: RE: [PATCH 5/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +static int ocrdma_inet6addr_event(struct notifier_block *,
> > + unsigned long, void *);
> > +
> > +static struct notifier_block ocrdma_inet6addr_notifier = {
> > + .notifier_call = ocrdma_inet6addr_event };
> > +
> > +int ocrdma_get_instance(void)
> > +{
> > + int instance = 0;
> > +
> > + /* Assign an unused number */
> > + if (!idr_pre_get(&ocrdma_dev_id, GFP_KERNEL))
> > + return -1;
> > + if (idr_get_new(&ocrdma_dev_id, NULL, &instance))
> > + return -1;
> > + return instance;
> > +}
> > +
> > +void ocrdma_get_guid(struct ocrdma_dev *dev, u8 *guid) {
> > + u8 mac_addr[6];
> > +
> > + memcpy(&mac_addr[0], &dev->nic_info.mac_addr[0], ETH_ALEN);
> > + guid[0] = mac_addr[0] ^ 2;
> > + guid[1] = mac_addr[1];
> > + guid[2] = mac_addr[2];
> > + guid[3] = 0xff;
> > + guid[4] = 0xfe;
> > + guid[5] = mac_addr[3];
> > + guid[6] = mac_addr[4];
> > + guid[7] = mac_addr[5];
> > +}
> > +
> > +static void ocrdma_build_sgid_mac(union ib_gid *sgid, unsigned char
> > *mac_addr,
> > + bool is_vlan, u16 vlan_id)
> > +{
> > + sgid->global.subnet_prefix = cpu_to_be64(0xfe80LL);
> > + sgid->raw[8] = mac_addr[0] ^ 2;
> > + sgid->raw[9] = mac_addr[1];
> > + sgid->raw[10] = mac_addr[2];
> > + if (is_vlan) {
> > + sgid->raw[11] = vlan_id >> 8;
> > + sgid->raw[12] = vlan_id & 0xff;
> > + } else {
> > + sgid->raw[11] = 0xff;
> > + sgid->raw[12] = 0xfe;
> > + }
> > + sgid->raw[13] = mac_addr[3];
> > + sgid->raw[14] = mac_addr[4];
> > + sgid->raw[15] = mac_addr[5];
> > +}
> > +
> > +static void ocrdma_add_sgid(struct ocrdma_dev *dev, unsigned char
> *mac_addr,
> > + bool is_vlan, u16 vlan_id)
> > +{
> > + int i;
> > + bool found = false;
> > + union ib_gid new_sgid;
> > + int free_idx = OCRDMA_MAX_SGID;
> > + unsigned long flags;
> > +
> > + memset(&ocrdma_zero_sgid, 0, sizeof(union ib_gid));
>
> ocrdma_zero_sgid should already be zeroed
>
Yes. Removed memset().
> Actually, can't we either use in6addr_any instead? I see that the mlx4 driver
> also has a zero gid. So, if we don't want to overload the use of in6addr_any,
> we can define gid_zero in ib_core.
o.k. once its defined in ib_core, will remove this and provide a subsequent
patch. It will be easier to handle it.
>
> > +
> > + ocrdma_build_sgid_mac(&new_sgid, mac_addr, is_vlan, vlan_id);
> > +
> > + spin_lock_irqsave(&dev->sgid_lock, flags);
> > + for (i = 0; i < OCRDMA_MAX_SGID; i++) {
> > + if (!memcmp(&dev->sgid_tbl[i], &ocrdma_zero_sgid,
> > + sizeof(union ib_gid))) {
> > + /* found free entry */
> > + if (!found) {
> > + free_idx = i;
> > + found = true;
> > + break;
> > + }
> > + } else if (!memcmp(&dev->sgid_tbl[i], &new_sgid,
> > + sizeof(union ib_gid))) {
> > + /* entry already present, no addition is required. */
> > + spin_unlock_irqrestore(&dev->sgid_lock, flags);
> > + return;
> > + }
> > + }
> > + /* if entry doesn't exist and if table has some space, add entry */
> > + if (found)
> > + memcpy(&dev->sgid_tbl[free_idx], &new_sgid,
> > + sizeof(union ib_gid));
>
> can move memcpy into for loop and remove found flag
>
Yes, removed.
> > + spin_unlock_irqrestore(&dev->sgid_lock, flags); }
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
Inline.
> -Original Message-
> From: Hefty, Sean [mailto:sean.he...@intel.com]
> Sent: Thursday, March 22, 2012 5:50 AM
> To: Pandit, Parav; linux-rdma@vger.kernel.org
> Subject: RE: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +static inline void *ocrdma_get_eqe(struct ocrdma_eq *eq) {
> > + return (u8 *)eq->q.va + (eq->q.tail * sizeof(struct ocrdma_eqe)); }
>
> casts from (void *) to (u8 *) are not needed. This occurs in multiple places.
Removed.
>
> > +enum ib_qp_state get_ibqp_state(enum ocrdma_qp_state qps) {
> > + switch (qps) {
> > + case OCRDMA_QPS_RST:
> > + return IB_QPS_RESET;
> > + case OCRDMA_QPS_INIT:
> > + return IB_QPS_INIT;
> > + case OCRDMA_QPS_RTR:
> > + return IB_QPS_RTR;
> > + case OCRDMA_QPS_RTS:
> > + return IB_QPS_RTS;
> > + case OCRDMA_QPS_SQD:
> > + case OCRDMA_QPS_SQ_DRAINING:
> > + return IB_QPS_SQD;
> > + case OCRDMA_QPS_SQE:
> > + return IB_QPS_SQE;
> > + case OCRDMA_QPS_ERR:
> > + return IB_QPS_ERR;
> > + };
> > + return IB_QPS_ERR;
> > +}
> > +
> > +enum ocrdma_qp_state get_ocrdma_qp_state(enum ib_qp_state qps) {
> > + switch (qps) {
> > + case IB_QPS_RESET:
> > + return OCRDMA_QPS_RST;
> > + case IB_QPS_INIT:
> > + return OCRDMA_QPS_INIT;
> > + case IB_QPS_RTR:
> > + return OCRDMA_QPS_RTR;
> > + case IB_QPS_RTS:
> > + return OCRDMA_QPS_RTS;
> > + case IB_QPS_SQD:
> > + return OCRDMA_QPS_SQD;
> > + case IB_QPS_SQE:
> > + return OCRDMA_QPS_SQE;
> > + case IB_QPS_ERR:
> > + return OCRDMA_QPS_ERR;
> > + };
> > + return OCRDMA_QPS_ERR;
> > +}
> > +
> > +static int ocrdma_get_mbx_errno(u32 status) {
> > + int err_num = -EFAULT;
>
> no need to initialize, since it's set in all cases
>
Removed.
> > + u8 mbox_status = (status & OCRDMA_MBX_RSP_STATUS_MASK) >>
> > + OCRDMA_MBX_RSP_STATUS_SHIFT;
> > + u8 add_status = (status & OCRDMA_MBX_RSP_ASTATUS_MASK) >>
> > +
> OCRDMA_MBX_RSP_ASTATUS_SHIFT;
> > +
> > + switch (mbox_status) {
> > + case OCRDMA_MBX_STATUS_OOR:
> > + case OCRDMA_MBX_STATUS_MAX_QP_EXCEEDS:
> > + err_num = -EAGAIN;
> > + break;
> > +
> > + case OCRDMA_MBX_STATUS_INVALID_PD:
> > + case OCRDMA_MBX_STATUS_INVALID_CQ:
> > + case OCRDMA_MBX_STATUS_INVALID_SRQ_ID:
> > + case OCRDMA_MBX_STATUS_INVALID_QP:
> > + case OCRDMA_MBX_STATUS_INVALID_CHANGE:
> > + case OCRDMA_MBX_STATUS_MTU_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_INVALID_RNR_NAK_TIMER:
> > + case OCRDMA_MBX_STATUS_PKEY_INDEX_INVALID:
> > + case OCRDMA_MBX_STATUS_PKEY_INDEX_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_ILLEGAL_FIELD:
> > + case OCRDMA_MBX_STATUS_INVALID_PBL_ENTRY:
> > + case OCRDMA_MBX_STATUS_INVALID_LKEY:
> > + case OCRDMA_MBX_STATUS_INVALID_VA:
> > + case OCRDMA_MBX_STATUS_INVALID_LENGTH:
> > + case OCRDMA_MBX_STATUS_INVALID_FBO:
> > + case OCRDMA_MBX_STATUS_INVALID_ACC_RIGHTS:
> > + case OCRDMA_MBX_STATUS_INVALID_PBE_SIZE:
> > + case OCRDMA_MBX_STATUS_ATOMIC_OPS_UNSUP:
> > + case OCRDMA_MBX_STATUS_SRQ_ERROR:
> > + case OCRDMA_MBX_STATUS_SRQ_SIZE_UNDERUNS:
> > + err_num = -EINVAL;
> > + break;
> > +
> > + case OCRDMA_MBX_STATUS_PD_INUSE:
> > + case OCRDMA_MBX_STATUS_QP_BOUND:
> > + case OCRDMA_MBX_STATUS_MW_STILL_BOUND:
> > + case OCRDMA_MBX_STATUS_MW_BOUND:
> > + err_num = -EBUSY;
> > + break;
> > +
> > + case OCRDMA_MBX_STATUS_RECVQ_RQE_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_SGE_RECV_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_RQE_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_SRQ_LIMIT_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_ORD_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_IRD_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_SENDQ_WQE_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_SGE_SEND_EXCEEDS:
> > + case OCRDMA_MBX_STATUS_SGE_WRITE_EXCEEDS:
> > + err_num = -ENOBUFS;
> > + break;
> > +
> > + case OCRDMA_MBX_STATUS_FAILED:
> > + switch (add_status) {
> > + case
> OCRDMA_MBX_ADDI_STATUS_INSUFFICIENT_RESOURCES:
> > + err_num = -EAGAIN;
> > + break;
> > + }
> > + default:
> > + err_num = -EFAULT;
> > + }
> > + return err_num;
> > +}
> > +
> > +static int ocrdma_get_mbx_cqe_errno(u16 cqe_status) {
> > + int err_num = -EINVAL;
>
> no need to initialize
>
> > + switch (cqe_status) {
> > + case OCRDMA_MBX_CQE_STATUS_INSUFFICIENT_PRIVILEDGES:
> > + err_num = -EPERM;
> > + break;
> > + case OCRDMA_MBX_CQE_STATUS_INVALID_PARAMETER:
> > + err_num = -EINVAL;
> > + break;
> > + case OCRDMA_MBX_CQE_STATUS_INSUFFICIENT_RESOURCES:
> > + case OCRDMA_MBX_CQE_STATUS_QUEUE_FLUSHING:
> > + err_num = -EAGAIN;
> > + break;
> > + case OCRDMA_MBX_CQE_STATUS_DMA_FAILED:
> > + err_num = -EIO;
> > +
RE: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Thursday, March 22, 2012 1:02 AM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> On Wed, Mar 21, 2012 at 12:09 PM, wrote:
> > Yes. Driver needs to put QP to flush state. So that appropriate CQEs can be
> returned during poll_cq() phase.
> > So state machine is implemented above.
>
> Couldn't you just write
>
> if (ib_modify_qp_is_ok(...)) {
> if (new_state == OCRDMA_QPS_ERR)
> ocrdma_flush_qp(qp);
> } else {
> status = -EINVAL;
> }
>
> and save about 100 lines of code?
>
Yes, this can be done. This is one path.
Another path is async_event coming from adapter. So I still need
qp_state_machine function but as you suggested, I'll remove the states and will
have invoke flush_qp() on error.
> - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Wednesday, March 21, 2012 10:13 PM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +struct ib_pd *ocrdma_alloc_pd(struct ib_device *ibdev,
> > + struct ib_ucontext *context,
> > + struct ib_udata *udata) {
> > + struct ocrdma_dev *dev = get_ocrdma_dev(ibdev);
> > + struct ocrdma_pd *pd;
> > + int status;
> > +
> > + pd = kzalloc(sizeof(*pd), GFP_KERNEL);
> > + if (!pd)
> > + return ERR_PTR(-ENOMEM);
> > + pd->dev = dev;
> > + if (udata && context) {
> > + pd->dpp_enabled = (dev->nic_info.dev_family ==
> > + OCRDMA_GEN2_FAMILY) ? true :
> > + false;
>
> Writing
>
> () ? true : false
>
> is pretty silly, since it's just an obfuscated way of writing
>
>
>
> IOW, you can just write
>
> pd->dpp_enabled = (dev->nic_info.dev_family ==
> OCRDMA_GEN2_FAMILY);
>
>
> > +int ocrdma_dealloc_pd(struct ib_pd *ibpd) {
> > + struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
> > + struct ocrdma_dev *dev = pd->dev;
> > + int status;
> > + u64 usr_db;
> > +
> > + if (atomic_read(&pd->use_cnt)) {
> > + ocrdma_err("%s(%d) pd=0x%x is in use.\n",
> > + __func__, dev->id, pd->id);
> > + status = -EFAULT;
> > + goto dealloc_err;
> > + }
>
> all of the use_cnt tracking in this driver seems to duplicate what the rdma
> midlayer already does... is there any reason we need that in the low-level
> hardware driver too, or can we just get rid of the various use_cnt members?
This use_cnt can be removed from low-level hardware driver. I'll remove it.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Wednesday, March 21, 2012 10:04 PM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +int ocrdma_qp_state_machine(struct ocrdma_qp *qp, enum ib_qp_state
> > +new_ib_state,
> > + enum ib_qp_state *old_ib_state) {
> > + unsigned long flags;
> > + int status = 0;
> > + enum ocrdma_qp_state new_state;
> > + new_state = get_ocrdma_qp_state(new_ib_state);
> > +
> > + /* sync with wqe and rqe posting */
> > + spin_lock_irqsave(&qp->q_lock, flags);
> > +
> > + if (old_ib_state)
> > + *old_ib_state = get_ibqp_state(qp->state);
> > + if (new_state == qp->state) {
> > + spin_unlock_irqrestore(&qp->q_lock, flags);
> > + return 1;
> > + }
> > +
> > + switch (qp->state) {
> > + case OCRDMA_QPS_RST:
> > + switch (new_state) {
> > + case OCRDMA_QPS_RST:
> > + case OCRDMA_QPS_INIT:
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + case OCRDMA_QPS_INIT:
> > + /* qps: INIT->XXX */
> > + switch (new_state) {
> > + case OCRDMA_QPS_INIT:
> > + case OCRDMA_QPS_RTR:
> > + break;
> > + case OCRDMA_QPS_ERR:
> > + ocrdma_flush_qp(qp);
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + case OCRDMA_QPS_RTR:
> > + /* qps: RTS->XXX */
> > + switch (new_state) {
> > + case OCRDMA_QPS_RTS:
> > + break;
> > + case OCRDMA_QPS_ERR:
> > + ocrdma_flush_qp(qp);
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + case OCRDMA_QPS_RTS:
> > + /* qps: RTS->XXX */
> > + switch (new_state) {
> > + case OCRDMA_QPS_SQD:
> > + case OCRDMA_QPS_SQE:
> > + break;
> > + case OCRDMA_QPS_ERR:
> > + ocrdma_flush_qp(qp);
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + case OCRDMA_QPS_SQD:
> > + /* qps: SQD->XXX */
> > + switch (new_state) {
> > + case OCRDMA_QPS_RTS:
> > + case OCRDMA_QPS_SQE:
> > + case OCRDMA_QPS_ERR:
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + case OCRDMA_QPS_SQE:
> > + switch (new_state) {
> > + case OCRDMA_QPS_RTS:
> > + case OCRDMA_QPS_ERR:
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + case OCRDMA_QPS_ERR:
> > + /* qps: ERR->XXX */
> > + switch (new_state) {
> > + case OCRDMA_QPS_RST:
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + break;
> > + default:
> > + status = -EINVAL;
> > + break;
> > + };
> > + if (!status)
> > + qp->state = new_state;
> > +
> > + spin_unlock_irqrestore(&qp->q_lock, flags);
> > + return status;
> > +}
>
> The switch statement here seems to largely reimpliment
> ib_modify_qp_is_ok() (which is exported from the rdma midlayer). Is there
> some reason that doesn't work for your driver? I'd rather fix / generalize
> the
> core helper function instead of having something mostly duplicate in a
> hardware driver.
Yes. Driver needs to put QP to flush state. So that appropriate CQEs can be
returned during poll_cq() phase.
So state machine is implemented above.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 3/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Wednesday, March 21, 2012 9:56 PM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 3/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +/* mailbox cmd response */
> > +struct ocrdma_mbx_rsp {
> > + u32 subsys_op;
> > + u32 status;
> > + u32 rsp_len;
> > + u32 add_rsp_len;
> > +} __packed;
>
> ...similar comments about only using __packed where you really need it...
This pack is required as it is shared with hardware and need to be of 16 bytes
for 32 and 64 bit architecture. Do not wanted to take risk of different
compiler versions. So keeping it packed.
>
> > +#define is_cqe_valid(cq, cqe) \
> > + (((le32_to_cpu(cqe->flags_status_srcqpn) & OCRDMA_CQE_VALID)\
> > + == cq->phase) ? 1 : 0)
> > +#define is_cqe_for_sq(cqe) \
> > + ((le32_to_cpu(cqe->flags_status_srcqpn) & OCRDMA_CQE_QTYPE) ?
> > +0 : 1) #define is_cqe_for_rq(cqe) \
> > + ((le32_to_cpu(cqe->flags_status_srcqpn) & OCRDMA_CQE_QTYPE) ?
> > +1 : 0) #define is_cqe_invalidated(cqe) \
> > + ((le32_to_cpu(cqe->flags_status_srcqpn) &
> > +OCRDMA_CQE_INVALIDATE) ? \
> > + 1 : 0)
> > +#define is_cqe_imm(cqe) \
> > + ((le32_to_cpu(cqe->flags_status_srcqpn) & OCRDMA_CQE_IMM) ? 1
> > +: 0) #define is_cqe_wr_imm(cqe) \
> > + ((le32_to_cpu(cqe->flags_status_srcqpn) &
> > +OCRDMA_CQE_WRITE_IMM) ? 1 : 0)
>
> ...similar comment about using readable typesafe inline functions instead of
> macros...
Yes, I'll change to inline function.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
I see couple of comments on rsvd words.
They were primarily not introduced for alignment. But there are other new
features that we will be adding with new set of hardware and firmware updates.
I don't want to change the user-kernel interface at such stage by modifying the
size of the structure.
For some features its under testing stage internally.
So once its ready rsvd will be replaced with actual element.
This will avoid abi compatibility issues between library and driver.
I'll consider alignment macro too so that compiler related byte alignment
access issue also gets resolved.
Parav
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Wednesday, March 21, 2012 9:50 PM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> On Tue, Mar 20, 2012 at 3:39 PM, wrote:
> > From: Parav Pandit
> >
> > - Header file for userspace library and kernel driver interface.
>
> > +struct ocrdma_alloc_ucontext_resp {
> > + u32 dev_id;
> > + u32 wqe_size;
> > + u32 max_inline_data;
> > + u32 dpp_wqe_size;
> > + u64 ah_tbl_page;
> > + u32 ah_tbl_len;
> > + u32 rsvd;
> > + u8 fw_ver[32];
> > + u32 rqe_size;
> > + u64 rsvd1;
> > +} __packed;
>
> If I'm reading this correctly, you have the 8-byte rsvd1 member at an offset
> only aligned to 4 bytes, because of the __packed directive. It would be much
> better to have these structures laid out so they are naturally the same on
> both 32-bit and 64-bit ABIs, and get rid of the __packed directive, which
> wrecks gcc code generation in some cases.
>
> In this particular case, it seems you could just move rqe_size into the slot
> where rsvd is, and get rid of rsvd1?
>
> > +/* user kernel communication data structures. */ struct
> > +ocrdma_alloc_pd_ureq {
> > + u64 rsvd1;
> > +} __packed;
>
> Similar comment -- __packed is silly for a structure with one reserved
> member (and which you don't seem to use anywhere)... why not just delete
> this struct?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Wednesday, March 21, 2012 9:44 PM
> To: Pandit, Parav
> Cc: linux-rdma@vger.kernel.org; net...@vger.kernel.org
> Subject: Re: [PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> > +#define ocrdma_err(format, arg...) printk(KERN_ERR format, ##arg)
>
> I think you'd be better off using pr_err() rather than defining your own
> macro.
o.k. I'll change it.
>
>
> > +struct ocrdma_cq {
> > + struct ib_cq ibcq;
> > + struct ocrdma_dev *dev;
> > + struct ocrdma_cqe *va;
> > + u32 phase;
> > + u32 getp; /* pointer to pending wrs to
> > + * return to stack, wrap arounds
> > + * at max_hw_cqe
> > + */
> > + u32 max_hw_cqe;
> > + bool phase_change;
> > + bool armed, solicited;
> > + bool arm_needed;
> > +
> > + spinlock_t cq_lock cacheline_aligned; /* provide
> > + synchronization
> > + * to cq polling
> > + */
> > + /* syncronizes cq completion handler invoked from multiple
> > + context */
> > + spinlock_t comp_handler_lock cacheline_aligned;
>
> You have quite a few of these alignment directives in the middle of
> structures.
> Have you measured that leaving all these gaps gives a reall performance
> boost?
>
>From beginning its cacheline aligned. So didn't tested it without it, but yes
>this is considered. I'll be shifting other widely used elements before the
>lock so that hole is smaller.
> > + u16 id;
> > + u16 eqn;
> > +
> > + struct ocrdma_ucontext *ucontext;
> > + dma_addr_t pa;
> > + u32 len;
> > + atomic_t use_cnt;
> > +
> > + /* head of all qp's sq and rq for which cqes need to be
> > +flushed
> > + * by the software.
> > + */
> > + struct list_head sq_head, rq_head; };
>
>
> > +#define OCRDMA_GET_NUM_POSTED_SHIFT_VAL(qp) \
> > + (((qp->dev->nic_info.dev_family == OCRDMA_GEN2_FAMILY) && \
> > + (qp->id < 64)) ? 24 : 16)
>
> In general it's better to use inline functions when possible instead of
> macros,
> which are less type-safe and harder to read.
o.k. I'll change it.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
Inline.
> -Original Message-
> From: Roland Dreier [mailto:rol...@purestorage.com]
> Sent: Wednesday, March 21, 2012 11:27 PM
> To: frank zago
> Cc: Pandit, Parav; linux-rdma@vger.kernel.org
> Subject: Re: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA
> adapter
>
> On Wed, Mar 21, 2012 at 10:42 AM, frank zago
> wrote:
> > On 03/20/2012 05:39 PM, parav.pan...@emulex.com wrote:
> >> +struct ib_mr *ocrdma_get_dma_mr(struct ib_pd *ibpd, int acc) {
> >> + struct ocrdma_mr *mr;
> >> +
> >> + mr = ocrdma_alloc_lkey(ibpd, acc, 0,
> >> + OCRDMA_ADDR_CHECK_DISABLE);
> >> + if (!mr)
> >> + return ERR_PTR(-ENOMEM);
> >
> > ocrdma_alloc_lkey does not return NULL on error.
>
I'll fix this part.
> Good catch! Even more to the point, ocrdma_alloc_lkey() doesn't return a
> struct ocrdma_mr* on success. So this function is totally broken.
>
It does returns ocrdma_mr* on success. Why do you think it doesn't return?
Below is the snippet.
status = ocrdma_mbx_alloc_lkey(dev, &mr->hwmr, pd->id, addr_check);
if (status) {
kfree(mr);
return ERR_PTR(-ENOMEM);
}
mr->pd = pd;
atomic_inc(&pd->use_cnt);
mr->ibmr.lkey = mr->hwmr.lkey;
if (mr->hwmr.remote_wr || mr->hwmr.remote_rd)
mr->ibmr.rkey = mr->hwmr.lkey;
return mr;
> - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 8/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit - build files for building ocrdma driver Signed-off-by: Parav Pandit --- drivers/infiniband/hw/ocrdma/Kconfig |8 drivers/infiniband/hw/ocrdma/Makefile |5 + 2 files changed, 13 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/hw/ocrdma/Kconfig create mode 100644 drivers/infiniband/hw/ocrdma/Makefile diff --git a/drivers/infiniband/hw/ocrdma/Kconfig b/drivers/infiniband/hw/ocrdma/Kconfig new file mode 100644 index 000..cf99342 --- /dev/null +++ b/drivers/infiniband/hw/ocrdma/Kconfig @@ -0,0 +1,8 @@ +config INFINIBAND_OCRDMA + tristate "Emulex One Connect HCA support" + depends on ETHERNET && NETDEVICES && PCI + select NET_VENDOR_EMULEX + select BE2NET + ---help--- + This driver provides low-level InfiniBand over Ethernet + support for Emulex One Connect host channel adapters (HCAs). diff --git a/drivers/infiniband/hw/ocrdma/Makefile b/drivers/infiniband/hw/ocrdma/Makefile new file mode 100644 index 000..06a5bed --- /dev/null +++ b/drivers/infiniband/hw/ocrdma/Makefile @@ -0,0 +1,5 @@ +ccflags-y := -Idrivers/net/ethernet/emulex/benet + +obj-$(CONFIG_INFINIBAND_OCRDMA)+= ocrdma.o + +ocrdma-y :=ocrdma_main.o ocrdma_verbs.o ocrdma_hw.o ocrdma_ah.o -- 1.6.0.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 9/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit - top level build files to build ocrdma driver. Signed-off-by: Parav Pandit --- drivers/infiniband/Kconfig |1 + drivers/infiniband/Makefile |1 + 2 files changed, 2 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig index eb0add3..a0f29c1 100644 --- a/drivers/infiniband/Kconfig +++ b/drivers/infiniband/Kconfig @@ -51,6 +51,7 @@ source "drivers/infiniband/hw/cxgb3/Kconfig" source "drivers/infiniband/hw/cxgb4/Kconfig" source "drivers/infiniband/hw/mlx4/Kconfig" source "drivers/infiniband/hw/nes/Kconfig" +source "drivers/infiniband/hw/ocrdma/Kconfig" source "drivers/infiniband/ulp/ipoib/Kconfig" diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile index a3b2d8e..bf846a1 100644 --- a/drivers/infiniband/Makefile +++ b/drivers/infiniband/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_INFINIBAND_CXGB3) += hw/cxgb3/ obj-$(CONFIG_INFINIBAND_CXGB4) += hw/cxgb4/ obj-$(CONFIG_MLX4_INFINIBAND) += hw/mlx4/ obj-$(CONFIG_INFINIBAND_NES) += hw/nes/ +obj-$(CONFIG_INFINIBAND_OCRDMA)+= hw/ocrdma/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ obj-$(CONFIG_INFINIBAND_SRPT) += ulp/srpt/ -- 1.6.0.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 7/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit
- address handle specific handling.
Signed-off-by: Parav Pandit
---
drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 172 ++
drivers/infiniband/hw/ocrdma/ocrdma_ah.h | 42 +++
2 files changed, 214 insertions(+), 0 deletions(-)
create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_ah.c
create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_ah.h
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
new file mode 100644
index 000..a877a8e
--- /dev/null
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
@@ -0,0 +1,172 @@
+/***
+ * This file is part of the Emulex RoCE Device Driver for *
+ * RoCE (RDMA over Converged Ethernet) adapters. *
+ * Copyright (C) 2008-2012 Emulex. All rights reserved.*
+ * EMULEX and SLI are trademarks of Emulex.*
+ * www.emulex.com *
+ * *
+ * This program is free software; you can redistribute it and/or *
+ * modify it under the terms of version 2 of the GNU General *
+ * Public License as published by the Free Software Foundation.*
+ * This program is distributed in the hope that it will be useful. *
+ * ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND *
+ * WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, *
+ * FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE *
+ * DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD *
+ * TO BE LEGALLY INVALID. See the GNU General Public License for *
+ * more details, a copy of which can be found in the file COPYING *
+ * included with this package. *
+ *
+ * Contact Information:
+ * linux-driv...@emulex.com
+ *
+ * Emulex
+ * Susan Street
+ * Costa Mesa, CA 92626
+ ***/
+
+#include
+#include
+
+#include
+#include
+
+#include "ocrdma.h"
+#include "ocrdma_verbs.h"
+#include "ocrdma_ah.h"
+#include "ocrdma_hw.h"
+
+static inline int set_av_attr(struct ocrdma_ah *ah,
+ struct ib_ah_attr *attr, int pdid)
+{
+ int status = 0;
+ u16 vlan_tag; bool vlan_enabled = false;
+ struct ocrdma_dev *dev = ah->dev;
+ struct ocrdma_eth_vlan eth;
+ struct ocrdma_grh grh;
+ int eth_sz;
+
+ memset(ð, 0, sizeof(eth));
+ memset(&grh, 0, sizeof(grh));
+
+ ah->sgid_index = attr->grh.sgid_index;
+
+ vlan_tag = rdma_get_vlan_id(&attr->grh.dgid);
+ if (vlan_tag && (vlan_tag < 0x1000)) {
+ eth.eth_type = cpu_to_be16(0x8100);
+ eth.roce_eth_type = cpu_to_be16(OCRDMA_ROCE_ETH_TYPE);
+ vlan_tag |= (attr->sl & 7) << 13;
+ eth.vlan_tag = cpu_to_be16(vlan_tag);
+ eth_sz = sizeof(struct ocrdma_eth_vlan);
+ vlan_enabled = true;
+ } else {
+ eth.eth_type = cpu_to_be16(OCRDMA_ROCE_ETH_TYPE);
+ eth_sz = sizeof(struct ocrdma_eth_basic);
+ }
+ memcpy(ð.smac[0], &dev->nic_info.mac_addr[0], ETH_ALEN);
+ status = ocrdma_resolve_dgid(dev, &attr->grh.dgid, ð.dmac[0]);
+ if (status)
+ return status;
+ status = ocrdma_query_gid(&dev->ibdev, 1, attr->grh.sgid_index,
+ (union ib_gid *)&grh.sgid[0]);
+ if (status)
+ return status;
+
+ grh.tclass_flow = cpu_to_be32((6 << 28) |
+ (attr->grh.traffic_class << 24) |
+ attr->grh.flow_label);
+ /* 0x1b is next header value in GRH */
+ grh.pdid_hoplimit = cpu_to_be32((pdid << 16) |
+ (0x1b << 8) | attr->grh.hop_limit);
+
+ memcpy(&grh.dgid[0], attr->grh.dgid.raw, sizeof(attr->grh.dgid.raw));
+ memcpy(&ah->av->eth_hdr, ð, eth_sz);
+ memcpy((u8 *)ah->av + eth_sz, &grh, sizeof(struct ocrdma_grh));
+ if (vlan_enabled)
+ ah->av->valid |= OCRDMA_AV_VLAN_VALID;
+ return status;
+}
+
+struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
+{
+ u32 *ahid_addr;
+ int status;
+ struct ocrdma_ah *ah;
+ struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
+ struct ocrdma_dev *dev = pd->dev;
+
+ if (!(attr->ah_flags & IB_AH_GRH))
+ return ERR_PTR(-EINVAL);
+
+ ah = kzalloc(sizeof *ah, GFP_ATOMIC);
+ if (!ah)
+ return ERR_PTR(-ENOMEM);
+ ah->dev = pd->dev;
+
+ status = ocrdma_alloc_av(dev, ah);
+ if (status)
+ goto av_err;
+ status = set_av_attr(ah, attr, pd->id);
+ if (status)
+ goto av_conf_err;
+
+ /* if pd is for the user process, pass the ah_id to user space */
+ if ((pd->uctx)
[PATCH 5/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit
Signed-off-by: Parav Pandit
---
drivers/infiniband/hw/ocrdma/ocrdma_main.c | 558
1 files changed, 558 insertions(+), 0 deletions(-)
create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_main.c
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_main.c
b/drivers/infiniband/hw/ocrdma/ocrdma_main.c
new file mode 100644
index 000..8aa3416
--- /dev/null
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_main.c
@@ -0,0 +1,558 @@
+/***
+ * This file is part of the Emulex RoCE Device Driver for *
+ * RoCE (RDMA over Converged Ethernet) adapters. *
+ * Copyright (C) 2008-2012 Emulex. All rights reserved.*
+ * EMULEX and SLI are trademarks of Emulex.*
+ * www.emulex.com *
+ * *
+ * This program is free software; you can redistribute it and/or *
+ * modify it under the terms of version 2 of the GNU General *
+ * Public License as published by the Free Software Foundation.*
+ * This program is distributed in the hope that it will be useful. *
+ * ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND *
+ * WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, *
+ * FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE *
+ * DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD *
+ * TO BE LEGALLY INVALID. See the GNU General Public License for *
+ * more details, a copy of which can be found in the file COPYING *
+ * included with this package. *
+ *
+ * Contact Information:
+ * linux-driv...@emulex.com
+ *
+ * Emulex
+ * Susan Street
+ * Costa Mesa, CA 92626
+ ***/
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include
+#include
+
+#include "ocrdma.h"
+#include "ocrdma_verbs.h"
+#include "ocrdma_ah.h"
+#include "be_roce.h"
+#include "ocrdma_hw.h"
+
+MODULE_VERSION(OCRDMA_ROCE_DEV_VERSION);
+MODULE_DESCRIPTION("Emulex RoCE HCA Driver");
+MODULE_AUTHOR("Emulex Corporation");
+MODULE_LICENSE("GPL");
+
+static LIST_HEAD(ocrdma_dev_list);
+static DEFINE_MUTEX(ocrdma_devlist_lock);
+static DEFINE_IDR(ocrdma_dev_id);
+
+static union ib_gid ocrdma_zero_sgid;
+static int ocrdma_inet6addr_event(struct notifier_block *,
+ unsigned long, void *);
+
+static struct notifier_block ocrdma_inet6addr_notifier = {
+ .notifier_call = ocrdma_inet6addr_event
+};
+
+int ocrdma_get_instance(void)
+{
+ int instance = 0;
+
+ /* Assign an unused number */
+ if (!idr_pre_get(&ocrdma_dev_id, GFP_KERNEL))
+ return -1;
+ if (idr_get_new(&ocrdma_dev_id, NULL, &instance))
+ return -1;
+ return instance;
+}
+
+void ocrdma_get_guid(struct ocrdma_dev *dev, u8 *guid)
+{
+ u8 mac_addr[6];
+
+ memcpy(&mac_addr[0], &dev->nic_info.mac_addr[0], ETH_ALEN);
+ guid[0] = mac_addr[0] ^ 2;
+ guid[1] = mac_addr[1];
+ guid[2] = mac_addr[2];
+ guid[3] = 0xff;
+ guid[4] = 0xfe;
+ guid[5] = mac_addr[3];
+ guid[6] = mac_addr[4];
+ guid[7] = mac_addr[5];
+}
+
+static void ocrdma_build_sgid_mac(union ib_gid *sgid, unsigned char *mac_addr,
+ bool is_vlan, u16 vlan_id)
+{
+ sgid->global.subnet_prefix = cpu_to_be64(0xfe80LL);
+ sgid->raw[8] = mac_addr[0] ^ 2;
+ sgid->raw[9] = mac_addr[1];
+ sgid->raw[10] = mac_addr[2];
+ if (is_vlan) {
+ sgid->raw[11] = vlan_id >> 8;
+ sgid->raw[12] = vlan_id & 0xff;
+ } else {
+ sgid->raw[11] = 0xff;
+ sgid->raw[12] = 0xfe;
+ }
+ sgid->raw[13] = mac_addr[3];
+ sgid->raw[14] = mac_addr[4];
+ sgid->raw[15] = mac_addr[5];
+}
+
+static void ocrdma_add_sgid(struct ocrdma_dev *dev, unsigned char *mac_addr,
+ bool is_vlan, u16 vlan_id)
+{
+ int i;
+ bool found = false;
+ union ib_gid new_sgid;
+ int free_idx = OCRDMA_MAX_SGID;
+ unsigned long flags;
+
+ memset(&ocrdma_zero_sgid, 0, sizeof(union ib_gid));
+
+ ocrdma_build_sgid_mac(&new_sgid, mac_addr, is_vlan, vlan_id);
+
+ spin_lock_irqsave(&dev->sgid_lock, flags);
+ for (i = 0; i < OCRDMA_MAX_SGID; i++) {
+ if (!memcmp(&dev->sgid_tbl[i], &ocrdma_zero_sgid,
+ sizeof(union ib_gid))) {
+ /* found free entry */
+ if (!found) {
+ free_idx = i;
+ found = true;
+ break;
+ }
+ } else if (!memcmp(&dev->sgid_tbl[i], &new_sgid,
+
[PATCH 3/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit
- Header file for driver-adapter interface.
Signed-off-by: Parav Pandit
---
drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 1672 +
1 files changed, 1672 insertions(+), 0 deletions(-)
create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_sli.h
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
new file mode 100644
index 000..7fd80cc
--- /dev/null
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
@@ -0,0 +1,1672 @@
+/***
+ * This file is part of the Emulex RoCE Device Driver for *
+ * RoCE (RDMA over Converged Ethernet) adapters. *
+ * Copyright (C) 2008-2012 Emulex. All rights reserved.*
+ * EMULEX and SLI are trademarks of Emulex.*
+ * www.emulex.com *
+ * *
+ * This program is free software; you can redistribute it and/or *
+ * modify it under the terms of version 2 of the GNU General *
+ * Public License as published by the Free Software Foundation.*
+ * This program is distributed in the hope that it will be useful. *
+ * ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND *
+ * WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, *
+ * FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE *
+ * DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD *
+ * TO BE LEGALLY INVALID. See the GNU General Public License for *
+ * more details, a copy of which can be found in the file COPYING *
+ * included with this package. *
+ *
+ * Contact Information:
+ * linux-driv...@emulex.com
+ *
+ * Emulex
+ * Susan Street
+ * Costa Mesa, CA 92626
+ ***/
+
+#ifndef __OCRDMA_SLI_H__
+#define __OCRDMA_SLI_H__
+
+#define Bit(_b) (1 << (_b))
+
+#define OCRDMA_GEN1_FAMILY 0xB
+#define OCRDMA_GEN2_FAMILY 0x2
+
+#define OCRDMA_SUBSYS_ROCE 10
+enum {
+ OCRDMA_CMD_QUERY_CONFIG = 1,
+ OCRDMA_CMD_ALLOC_PD,
+ OCRDMA_CMD_DEALLOC_PD,
+
+ OCRDMA_CMD_CREATE_AH_TBL,
+ OCRDMA_CMD_DELETE_AH_TBL,
+
+ OCRDMA_CMD_CREATE_QP,
+ OCRDMA_CMD_QUERY_QP,
+ OCRDMA_CMD_MODIFY_QP,
+ OCRDMA_CMD_DELETE_QP,
+
+ OCRDMA_CMD_RSVD1,
+ OCRDMA_CMD_ALLOC_LKEY,
+ OCRDMA_CMD_DEALLOC_LKEY,
+ OCRDMA_CMD_REGISTER_NSMR,
+ OCRDMA_CMD_REREGISTER_NSMR,
+ OCRDMA_CMD_REGISTER_NSMR_CONT,
+ OCRDMA_CMD_QUERY_NSMR,
+ OCRDMA_CMD_ALLOC_MW,
+ OCRDMA_CMD_QUERY_MW,
+
+ OCRDMA_CMD_CREATE_SRQ,
+ OCRDMA_CMD_QUERY_SRQ,
+ OCRDMA_CMD_MODIFY_SRQ,
+ OCRDMA_CMD_DELETE_SRQ,
+
+ OCRDMA_CMD_ATTACH_MCAST,
+ OCRDMA_CMD_DETACH_MCAST,
+
+ OCRDMA_CMD_MAX
+};
+
+#define OCRDMA_SUBSYS_COMMON 1
+enum {
+ OCRDMA_CMD_CREATE_CQ= 12,
+ OCRDMA_CMD_CREATE_EQ= 13,
+ OCRDMA_CMD_CREATE_MQ= 21,
+ OCRDMA_CMD_GET_FW_VER = 35,
+ OCRDMA_CMD_DELETE_MQ= 53,
+ OCRDMA_CMD_DELETE_CQ= 54,
+ OCRDMA_CMD_DELETE_EQ= 55,
+ OCRDMA_CMD_GET_FW_CONFIG= 58,
+ OCRDMA_CMD_CREATE_MQ_EXT= 90
+};
+
+enum {
+ QTYPE_EQ= 1,
+ QTYPE_CQ= 2,
+ QTYPE_MCCQ = 3
+};
+
+#define OCRDMA_MAX_SGID (8)
+
+#define OCRDMA_MAX_QP2048
+#define OCRDMA_MAX_CQ2048
+
+enum {
+ OCRDMA_DB_RQ_OFFSET = 0xE0,
+ OCRDMA_DB_GEN2_RQ1_OFFSET = 0x100,
+ OCRDMA_DB_GEN2_RQ2_OFFSET = 0xC0,
+ OCRDMA_DB_SQ_OFFSET = 0x60,
+ OCRDMA_DB_GEN2_SQ_OFFSET= 0x1C0,
+ OCRDMA_DB_SRQ_OFFSET= OCRDMA_DB_RQ_OFFSET,
+ OCRDMA_DB_GEN2_SRQ_OFFSET = OCRDMA_DB_GEN2_RQ1_OFFSET,
+ OCRDMA_DB_CQ_OFFSET = 0x120,
+ OCRDMA_DB_EQ_OFFSET = OCRDMA_DB_CQ_OFFSET,
+ OCRDMA_DB_MQ_OFFSET = 0x140
+};
+
+#define OCRDMA_DB_CQ_RING_ID_MASK 0x3FF /* bits 0 - 9 */
+#define OCRDMA_DB_CQ_RING_ID_EXT_MASK 0x0C00 /* bits 10-11 of qid at 12-11 */
+/* qid #2 msbits at 12-11 */
+#define OCRDMA_DB_CQ_RING_ID_EXT_MASK_SHIFT 0x1
+#define OCRDMA_DB_CQ_NUM_POPPED_SHIFT (16) /* bits 16 - 28 */
+/* Rearm bit */
+#define OCRDMA_DB_CQ_REARM_SHIFT(29) /* bit 29 */
+/* solicited bit */
+#define OCRDMA_DB_CQ_SOLICIT_SHIFT (31) /* bit 31 */
+
+#define OCRDMA_EQ_ID_MASK 0x1FF /* bits 0 - 8 */
+#define OCRDMA_EQ_ID_EXT_MASK 0x3e00 /* bits 9-13 */
+#define OCRDMA_EQ_ID_EXT_MASK_SHIFT(2) /* qid bits 9-13 at 11-15 */
+
+/* Clear the interrupt for this eq */
+#define OCRDMA_EQ_CLR_SHIFT(9) /* bit 9 */
+/* Must be 1 */
+#define OCRDMA_EQ_TYPE_SHIFT
[PATCH 2/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit
- Header file for userspace library and kernel driver interface.
Signed-off-by: Parav Pandit
---
drivers/infiniband/hw/ocrdma/ocrdma_abi.h | 134 +
1 files changed, 134 insertions(+), 0 deletions(-)
create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_abi.h
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_abi.h
b/drivers/infiniband/hw/ocrdma/ocrdma_abi.h
new file mode 100644
index 000..a411a4e
--- /dev/null
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_abi.h
@@ -0,0 +1,134 @@
+/***
+ * This file is part of the Emulex RoCE Device Driver for *
+ * RoCE (RDMA over Converged Ethernet) adapters. *
+ * Copyright (C) 2008-2012 Emulex. All rights reserved.*
+ * EMULEX and SLI are trademarks of Emulex.*
+ * www.emulex.com *
+ * *
+ * This program is free software; you can redistribute it and/or *
+ * modify it under the terms of version 2 of the GNU General *
+ * Public License as published by the Free Software Foundation.*
+ * This program is distributed in the hope that it will be useful. *
+ * ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND *
+ * WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, *
+ * FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE *
+ * DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD *
+ * TO BE LEGALLY INVALID. See the GNU General Public License for *
+ * more details, a copy of which can be found in the file COPYING *
+ * included with this package. *
+ *
+ * Contact Information:
+ * linux-driv...@emulex.com
+ *
+ * Emulex
+ * Susan Street
+ * Costa Mesa, CA 92626
+ ***/
+
+#ifndef __OCRDMA_ABI_H__
+#define __OCRDMA_ABI_H__
+
+struct ocrdma_alloc_ucontext_resp {
+ u32 dev_id;
+ u32 wqe_size;
+ u32 max_inline_data;
+ u32 dpp_wqe_size;
+ u64 ah_tbl_page;
+ u32 ah_tbl_len;
+ u32 rsvd;
+ u8 fw_ver[32];
+ u32 rqe_size;
+ u64 rsvd1;
+} __packed;
+
+/* user kernel communication data structures. */
+struct ocrdma_alloc_pd_ureq {
+ u64 rsvd1;
+} __packed;
+
+struct ocrdma_alloc_pd_uresp {
+ u32 id;
+ u32 dpp_enabled;
+ u32 dpp_page_addr_hi;
+ u32 dpp_page_addr_lo;
+ u64 rsvd1;
+} __packed;
+
+struct ocrdma_create_cq_ureq {
+ u32 dpp_cq;
+ u32 rsvd;
+} __packed;
+
+#define MAX_CQ_PAGES 8
+struct ocrdma_create_cq_uresp {
+ u32 cq_id;
+ u32 page_size;
+ u32 num_pages;
+ u32 max_hw_cqe;
+ u64 page_addr[MAX_CQ_PAGES];
+ u64 db_page_addr;
+ u32 db_page_size;
+ u32 phase_change;
+ u64 rsvd1;
+ u64 rsvd2;
+} __packed;
+
+#define MAX_QP_PAGES 8
+#define MAX_UD_AV_PAGES 8
+
+struct ocrdma_create_qp_ureq {
+ u8 enable_dpp_cq;
+ u8 rsvd;
+ u16 dpp_cq_id;
+ u32 rsvd1;
+};
+
+struct ocrdma_create_qp_uresp {
+ u16 qp_id;
+ u16 sq_dbid;
+ u16 rq_dbid;
+ u16 resv0;
+ u32 sq_page_size;
+ u32 rq_page_size;
+ u32 num_sq_pages;
+ u32 num_rq_pages;
+ u64 sq_page_addr[MAX_QP_PAGES];
+ u64 rq_page_addr[MAX_QP_PAGES];
+ u64 db_page_addr;
+ u32 db_page_size;
+ u32 dpp_credit;
+ u32 dpp_offset;
+ u32 rsvd1;
+ u32 num_wqe_allocated;
+ u32 num_rqe_allocated;
+ u32 free_wqe_delta;
+ u32 free_rqe_delta;
+ u32 db_sq_offset;
+ u32 db_rq_offset;
+ u32 db_shift;
+ u64 rsvd2;
+ u64 rsvd3;
+} __packed;
+
+struct ocrdma_create_srq_uresp {
+ u16 rq_dbid;
+ u16 resv0;
+ u32 resv1;
+
+ u32 rq_page_size;
+ u32 num_rq_pages;
+
+ u64 rq_page_addr[MAX_QP_PAGES];
+ u64 db_page_addr;
+
+ u32 db_page_size;
+ u32 num_rqe_allocated;
+ u32 db_rq_offset;
+ u32 db_shift;
+
+ u32 free_rqe_delta;
+ u32 rsvd2;
+ u64 rsvd3;
+} __packed;
+
+#endif /* __OCRDMA_ABI_H__ */
--
1.6.0.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/9] ocrdma: Driver for Emulex OneConnect RDMA adapter
From: Parav Pandit
- Header file for device and resource specific data structures.
Signed-off-by: Parav Pandit
---
drivers/infiniband/hw/ocrdma/ocrdma.h | 392 +
1 files changed, 392 insertions(+), 0 deletions(-)
create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma.h
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h
b/drivers/infiniband/hw/ocrdma/ocrdma.h
new file mode 100644
index 000..d7a44b8
--- /dev/null
+++ b/drivers/infiniband/hw/ocrdma/ocrdma.h
@@ -0,0 +1,392 @@
+/***
+ * This file is part of the Emulex RoCE Device Driver for *
+ * RoCE (RDMA over Converged Ethernet) adapters. *
+ * Copyright (C) 2008-2012 Emulex. All rights reserved.*
+ * EMULEX and SLI are trademarks of Emulex.*
+ * www.emulex.com *
+ * *
+ * This program is free software; you can redistribute it and/or *
+ * modify it under the terms of version 2 of the GNU General *
+ * Public License as published by the Free Software Foundation.*
+ * This program is distributed in the hope that it will be useful. *
+ * ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND *
+ * WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, *
+ * FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE *
+ * DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD *
+ * TO BE LEGALLY INVALID. See the GNU General Public License for *
+ * more details, a copy of which can be found in the file COPYING *
+ * included with this package. *
+ *
+ * Contact Information:
+ * linux-driv...@emulex.com
+ *
+ * Emulex
+ * Susan Street
+ * Costa Mesa, CA 92626
+ ***/
+
+#ifndef __OCRDMA_H__
+#define __OCRDMA_H__
+
+#include
+#include
+#include
+#include
+
+#include
+#include
+
+#include
+#include "ocrdma_sli.h"
+
+#define OCRDMA_ROCE_DEV_VERSION "1.0.0"
+#define OCRDMA_NODE_DESC "Emulex OneConnect RoCE HCA"
+
+#define ocrdma_err(format, arg...) printk(KERN_ERR format, ##arg)
+
+#define OCRDMA_MAX_AH 512
+
+#define OCRDMA_UVERBS(CMD_NAME) (1ull << IB_USER_VERBS_CMD_##CMD_NAME)
+
+struct ocrdma_dev_attr {
+ u8 fw_ver[32];
+ u32 vendor_id;
+ u32 device_id;
+ u16 max_pd;
+ u16 max_cq;
+ u16 max_cqe;
+ u16 max_qp;
+ u16 max_wqe;
+ u16 max_rqe;
+ u32 max_inline_data;
+ int max_send_sge;
+ int max_recv_sge;
+ int max_mr;
+ u64 max_mr_size;
+ u32 max_num_mr_pbl;
+ int max_fmr;
+ int max_map_per_fmr;
+ int max_pages_per_frmr;
+ u16 max_ord_per_qp;
+ u16 max_ird_per_qp;
+
+ int device_cap_flags;
+ u8 cq_overflow_detect;
+ u8 srq_supported;
+
+ u32 wqe_size;
+ u32 rqe_size;
+ u32 ird_page_size;
+ u8 local_ca_ack_delay;
+ u8 ird;
+ u8 num_ird_pages;
+};
+
+struct ocrdma_pbl {
+ void *va;
+ dma_addr_t pa;
+};
+
+struct ocrdma_queue_info {
+ void *va;
+ dma_addr_t dma;
+ u32 size;
+ u16 len;
+ u16 entry_size; /* Size of an element in the queue */
+ u16 id; /* qid, where to ring the doorbell. */
+ u16 head, tail;
+ bool created;
+ atomic_t used; /* Number of valid elements in the queue */
+};
+
+struct ocrdma_eq {
+ struct ocrdma_queue_info q;
+ u32 vector;
+ int cq_cnt;
+ struct ocrdma_dev *dev;
+ char irq_name[32];
+};
+
+struct ocrdma_mq {
+ struct ocrdma_queue_info sq;
+ struct ocrdma_queue_info cq;
+ bool rearm_cq;
+};
+
+struct mqe_ctx {
+ struct mutex lock; /* for serializing mailbox commands on MQ */
+ wait_queue_head_t cmd_wait;
+ u32 tag;
+ u16 cqe_status;
+ u16 ext_status;
+ bool cmd_done;
+};
+
+struct ocrdma_dev {
+ struct ib_device ibdev;
+ struct ocrdma_dev_attr attr;
+
+ struct mutex dev_lock; /* provides syncronise access to device data */
+ spinlock_t flush_q_lock cacheline_aligned;
+
+ struct ocrdma_cq **cq_tbl;
+ struct ocrdma_qp **qp_tbl;
+
+ struct ocrdma_eq meq;
+ struct ocrdma_eq *qp_eq_tbl;
+ int eq_cnt;
+ u16 base_eqid;
+ u16 max_eq;
+
+ union ib_gid *sgid_tbl;
+ /* provided synchronization to sgid table for
+* updating gid entries triggered by notifier.
+*/
+ spinlock_t sgid_lock;
+
+ int gsi_qp_created;
+ struct ocrdma_cq *gsi_sqcq;
+ struct ocrdma_cq *gsi_rqcq;
+
+ struct {
+ struct ocrdma_av *va;
+ dma_addr_t pa;
+ u32 size;
+ u32 num_ah;
+ /* provide synchronization f
[PATCH 0/9] ocrdma: Driver for Emulex OneConnect RDMA
From: Parav Pandit Emulex One Connect Adapter is RDMA (RoCE) capable multi-function PCI Express device. This driver patch enables RoCE support on such adapter. This ocrdma driver depends on be2net NIC driver. This patch depends on the previously submitted be2net NIC driver patch. Code organization: - ocrdma.h : driver header file. - ocrdma_main.c : driver registration with stack. - ocrdma_sli.h : driver-adapter interface definitions. - ocrdma_hw.*: hardware specific initialization, mailbox cmds. - ocrdma_verbs.* : verbs interface functionality. - ocrdma_ah.*: address handle related functionaliy. - ocrdma_abi.h : user space library interaction definitions. This patch is made against the current git tree. Please provide your review comments. I have resent this patch as previous bigger patch was dropped by vger.kernel.org. Thank you. Parav Pandit (9): ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter ocrdma: Driver for Emulex OneConnect RDMA adapter drivers/infiniband/Kconfig |1 + drivers/infiniband/Makefile |1 + drivers/infiniband/hw/ocrdma/Kconfig|8 + drivers/infiniband/hw/ocrdma/Makefile |5 + drivers/infiniband/hw/ocrdma/ocrdma.h | 392 drivers/infiniband/hw/ocrdma/ocrdma_abi.h | 134 ++ drivers/infiniband/hw/ocrdma/ocrdma_ah.c| 172 ++ drivers/infiniband/hw/ocrdma/ocrdma_ah.h| 42 + drivers/infiniband/hw/ocrdma/ocrdma_hw.c| 2640 +++ drivers/infiniband/hw/ocrdma/ocrdma_hw.h| 132 ++ drivers/infiniband/hw/ocrdma/ocrdma_main.c | 558 ++ drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 1672 + drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 2542 ++ drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 94 + 14 files changed, 8393 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/hw/ocrdma/Kconfig create mode 100644 drivers/infiniband/hw/ocrdma/Makefile create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_abi.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_ah.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_ah.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_hw.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_hw.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_main.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_sli.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_verbs.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_verbs.h -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/2] ocrdma: Driver for Emulex OneConnect RDMA device.
Ok. I'll resend it with smaller patches. Parav > -Original Message- > From: Roland Dreier [mailto:rol...@purestorage.com] > Sent: Tuesday, March 20, 2012 10:21 PM > To: Or Gerlitz > Cc: Pandit, Parav; linux-rdma@vger.kernel.org > Subject: Re: [PATCH 0/2] ocrdma: Driver for Emulex OneConnect RDMA > device. > > On Tue, Mar 20, 2012 at 9:42 AM, Or Gerlitz wrote: > > patch 1/2 didn't reach the list, if it happens to be > 100KB > > vger.kernel.org probably dropped it > > Parav, can you split the patch into a few pieces and resend? > > Thanks, > Roland -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] ocrdma: Driver for Emulex OneConnect RDMA device.
From: Parav Pandit - Added entries to build ocrdma driver. Signed-off-by: Parav Pandit --- drivers/infiniband/Kconfig |1 + drivers/infiniband/Makefile |1 + 2 files changed, 2 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig index eb0add3..a0f29c1 100644 --- a/drivers/infiniband/Kconfig +++ b/drivers/infiniband/Kconfig @@ -51,6 +51,7 @@ source "drivers/infiniband/hw/cxgb3/Kconfig" source "drivers/infiniband/hw/cxgb4/Kconfig" source "drivers/infiniband/hw/mlx4/Kconfig" source "drivers/infiniband/hw/nes/Kconfig" +source "drivers/infiniband/hw/ocrdma/Kconfig" source "drivers/infiniband/ulp/ipoib/Kconfig" diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile index a3b2d8e..bf846a1 100644 --- a/drivers/infiniband/Makefile +++ b/drivers/infiniband/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_INFINIBAND_CXGB3) += hw/cxgb3/ obj-$(CONFIG_INFINIBAND_CXGB4) += hw/cxgb4/ obj-$(CONFIG_MLX4_INFINIBAND) += hw/mlx4/ obj-$(CONFIG_INFINIBAND_NES) += hw/nes/ +obj-$(CONFIG_INFINIBAND_OCRDMA)+= hw/ocrdma/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ obj-$(CONFIG_INFINIBAND_SRPT) += ulp/srpt/ -- 1.6.0.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 0/2] ocrdma: Driver for Emulex OneConnect RDMA device.
From: Parav Pandit Emulex One Connect Adapter is RDMA (RoCE) capable multi-function PCI Express device. This driver patch enables RoCE support on such adapter. This ocrdma driver depends on be2net NIC driver. This patch depends on the previously submitted be2net NIC driver patch. Code organization: - ocrdma.h : driver header file. - ocrdma_main.c : driver registration with stack. - ocrdma_sli.h : driver-adapter interface definitions. - ocrdma_hw.*: hardware specific initialization, mailbox cmds. - ocrdma_verbs.* : verbs interface functionality. - ocrdma_ah.*: address handle related functionaliy. - ocrdma_abi.h : user space library interaction definitions. This patch is made against the current git tree. Please provide your review comments. Thank you. Parav Pandit (2): ocrdma: Driver for Emulex OneConnect RDMA device. ocrdma: Driver for Emulex OneConnect RDMA device. drivers/infiniband/Kconfig |1 + drivers/infiniband/Makefile |1 + drivers/infiniband/hw/ocrdma/Kconfig|8 + drivers/infiniband/hw/ocrdma/Makefile |5 + drivers/infiniband/hw/ocrdma/ocrdma.h | 392 drivers/infiniband/hw/ocrdma/ocrdma_abi.h | 134 ++ drivers/infiniband/hw/ocrdma/ocrdma_ah.c| 172 ++ drivers/infiniband/hw/ocrdma/ocrdma_ah.h| 42 + drivers/infiniband/hw/ocrdma/ocrdma_hw.c| 2640 +++ drivers/infiniband/hw/ocrdma/ocrdma_hw.h| 132 ++ drivers/infiniband/hw/ocrdma/ocrdma_main.c | 558 ++ drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 1672 + drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 2542 ++ drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 94 + 14 files changed, 8393 insertions(+), 0 deletions(-) create mode 100644 drivers/infiniband/hw/ocrdma/Kconfig create mode 100644 drivers/infiniband/hw/ocrdma/Makefile create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_abi.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_ah.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_ah.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_hw.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_hw.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_main.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_sli.h create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_verbs.c create mode 100644 drivers/infiniband/hw/ocrdma/ocrdma_verbs.h -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V1 1/2] be2net: Added function to issue mailbox cmd on MQ.
From: Parav Pandit
- Added generic function to issue mailbox cmd on MQ as export function.
- RoCE driver will use this before it setups its own MQ.
Signed-off-by: Parav Pandit
---
drivers/net/ethernet/emulex/benet/be_cmds.c | 39 +++
1 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/drivers/net/ethernet/emulex/benet/be_cmds.c
b/drivers/net/ethernet/emulex/benet/be_cmds.c
index 0fcb456..19037b0 100644
--- a/drivers/net/ethernet/emulex/benet/be_cmds.c
+++ b/drivers/net/ethernet/emulex/benet/be_cmds.c
@@ -15,6 +15,7 @@
* Costa Mesa, CA 92626
*/
+#include
#include "be.h"
#include "be_cmds.h"
@@ -2391,3 +2392,41 @@ err:
spin_unlock_bh(&adapter->mcc_lock);
return status;
}
+
+int be_roce_mcc_cmd(void *netdev_handle, void *wrb_payload,
+ int wrb_payload_size, u16 *cmd_status, u16 *ext_status)
+{
+ struct be_adapter *adapter = netdev_priv(netdev_handle);
+ struct be_mcc_wrb *wrb;
+ struct be_cmd_req_hdr *hdr = (struct be_cmd_req_hdr *) wrb_payload;
+ struct be_cmd_req_hdr *req;
+ struct be_cmd_resp_hdr *resp;
+ int status;
+
+ spin_lock_bh(&adapter->mcc_lock);
+
+ wrb = wrb_from_mccq(adapter);
+ if (!wrb) {
+ status = -EBUSY;
+ goto err;
+ }
+ req = embedded_payload(wrb);
+ resp = embedded_payload(wrb);
+
+ be_wrb_cmd_hdr_prepare(req, hdr->subsystem,
+ hdr->opcode, wrb_payload_size, wrb, NULL);
+ memcpy(req, wrb_payload, wrb_payload_size);
+ be_dws_cpu_to_le(req, wrb_payload_size);
+
+ status = be_mcc_notify_wait(adapter);
+ if (cmd_status)
+ *cmd_status = (status & 0x);
+ if (ext_status)
+ *ext_status = 0;
+ memcpy(wrb_payload, resp, sizeof(*resp) + resp->response_length);
+ be_dws_le_to_cpu(wrb_payload, sizeof(*resp) + resp->response_length);
+err:
+ spin_unlock_bh(&adapter->mcc_lock);
+ return status;
+}
+EXPORT_SYMBOL(be_roce_mcc_cmd);
--
1.6.0.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH V1 2/2] be2net: Added functionality to support RoCE driver
From: Parav Pandit
- Increased MSIX vectors by 5 for RoCE traffic.
- Added macro to check roce support on a device.
- Added device specific doorbell, msix vector fields shared with nic
functionality.
- Provides RoCE driver registration and deregistration functions.
- Added support functions which will be invoked on adapter
add/remove and port up/down events.
- Traverses through the list of adapters for invoking callback functions.
Signed-off-by: Parav Pandit
---
drivers/net/ethernet/emulex/benet/Makefile |2 +-
drivers/net/ethernet/emulex/benet/be.h | 38 ++-
drivers/net/ethernet/emulex/benet/be_cmds.h |1 +
drivers/net/ethernet/emulex/benet/be_hw.h |4 +-
drivers/net/ethernet/emulex/benet/be_main.c | 83 ++--
drivers/net/ethernet/emulex/benet/be_roce.c | 183 +++
drivers/net/ethernet/emulex/benet/be_roce.h | 75 +++
7 files changed, 370 insertions(+), 16 deletions(-)
create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.c
create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.h
diff --git a/drivers/net/ethernet/emulex/benet/Makefile
b/drivers/net/ethernet/emulex/benet/Makefile
index a60cd80..1a91b27 100644
--- a/drivers/net/ethernet/emulex/benet/Makefile
+++ b/drivers/net/ethernet/emulex/benet/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_BE2NET) += be2net.o
-be2net-y := be_main.o be_cmds.o be_ethtool.o
+be2net-y := be_main.o be_cmds.o be_ethtool.o be_roce.o
diff --git a/drivers/net/ethernet/emulex/benet/be.h
b/drivers/net/ethernet/emulex/benet/be.h
index cbdec25..fc094d9 100644
--- a/drivers/net/ethernet/emulex/benet/be.h
+++ b/drivers/net/ethernet/emulex/benet/be.h
@@ -32,6 +32,7 @@
#include
#include "be_hw.h"
+#include "be_roce.h"
#define DRV_VER"4.0.100u"
#define DRV_NAME "be2net"
@@ -92,7 +93,7 @@ static inline char *nic_name(struct pci_dev *pdev)
#define MAX_RSS_QS 4 /* BE limit is 4 queues/port */
#define MAX_RX_QS (MAX_RSS_QS + 1) /* RSS qs + 1 def Rx */
#define MAX_TX_QS 8
-#define BE_MAX_MSIX_VECTORS(MAX_RX_QS + 1)/* RX + TX */
+#define BE_MAX_MSIX_VECTORS(MAX_RX_QS + 1 + 5)/* RX + TX + 5 RoCE */
#define BE_NAPI_WEIGHT 64
#define MAX_RX_POSTBE_NAPI_WEIGHT /* Frags posted at a time */
#define RX_FRAGS_REFILL_WM (RX_Q_LEN - MAX_RX_POST)
@@ -320,6 +321,7 @@ struct be_adapter {
struct msix_entry msix_entries[BE_MAX_MSIX_VECTORS];
u32 num_msix_vec;
+ u32 num_eqs;
bool isr_registered;
/* TX Rings */
@@ -372,6 +374,16 @@ struct be_adapter {
u8 transceiver;
u8 autoneg;
u8 generation; /* BladeEngine ASIC generation */
+ u32 if_type;
+ struct {
+ u8 __iomem *base; /* Door Bell */
+ u32 size;
+ u32 total_size;
+ u64 io_addr;
+ } roce_db;
+ struct ocrdma_dev *ocrdma_dev;
+ struct list_head entry;
+
u32 flash_status;
struct completion flash_compl;
@@ -399,6 +411,10 @@ struct be_adapter {
#define lancer_chip(adapter) ((adapter->pdev->device == OC_DEVICE_ID3) || \
(adapter->pdev->device == OC_DEVICE_ID4))
+#define be_roce_supported(adapter) ((adapter->if_type == SLI_INTF_TYPE_3 || \
+ adapter->sli_family == SKYHAWK_SLI_FAMILY) && \
+ (adapter->function_mode & RDMA_ENABLED))
+
extern const struct ethtool_ops be_ethtool_ops;
#define msix_enabled(adapter) (adapter->num_msix_vec > 0)
@@ -539,9 +555,29 @@ static inline bool be_error(struct be_adapter *adapter)
return adapter->eeh_err || adapter->ue_detected || adapter->fw_timeout;
}
+
+static inline bool be_type_2_3(struct be_adapter *adapter)
+{
+ return (adapter->if_type == SLI_INTF_TYPE_2 ||
+ adapter->if_type == SLI_INTF_TYPE_3) ? true : false;
+}
+
extern void be_cq_notify(struct be_adapter *adapter, u16 qid, bool arm,
u16 num_popped);
extern void be_link_status_update(struct be_adapter *adapter, u8 link_status);
extern void be_parse_stats(struct be_adapter *adapter);
extern int be_load_fw(struct be_adapter *adapter, u8 *func);
+
+/*
+ * internal function to initialize-cleanup roce device.
+ */
+extern void be_roce_dev_add(struct be_adapter *);
+extern void be_roce_dev_remove(struct be_adapter *);
+
+/*
+ * internal function to open-close roce device during ifup-ifdown.
+ */
+extern void be_roce_dev_open(struct be_adapter *);
+extern void be_roce_dev_close(struct be_adapter *);
+
#endif /* BE_H */
diff --git a/drivers/net/ethernet/emulex/benet/be_cmds.h
b/drivers/net/ethernet/emulex/benet/be_cmds.h
index dca8924..d7ad03b 100644
--- a/drivers/net/ethernet/emulex/benet/be_cmds.h
+++ b/drivers/net/ethernet/emulex/benet/be_cmds.h
@@ -1057,6 +1057,7 @@ struct be_cmd_resp_modify
[PATCH V1 0/2] be2net: Added functionality to support RoCE driver
From: Parav Pandit This patch addresses all the review comments given by Francois Romieu & David Laight for past patch except be_roce_supported() macro as it breaks the modularity of be_cmds.x This patch adds functionality to support RoCE (RDMA over Ethernet) driver. - Detecting RoCE supported adapters and creating linked list of them. - Enabling 5 more MSIX vectors for RoCE functionality. - Calling registered callback functions of the RoCE driver whenever new RoCE capable device is added/removed. - Notifying events to RoCE driver when interface is up or down. - Provides device specific details to RoCE driver for each roce device. - Provides low level mailbox command to be issued by RoCE driver before it can have it own MQ. Parav Pandit (2): be2net: Added function to issue mailbox cmd on MQ. be2net: Added functionality to support RoCE driver drivers/net/ethernet/emulex/benet/Makefile |2 +- drivers/net/ethernet/emulex/benet/be.h | 38 ++- drivers/net/ethernet/emulex/benet/be_cmds.c | 39 ++ drivers/net/ethernet/emulex/benet/be_cmds.h |1 + drivers/net/ethernet/emulex/benet/be_hw.h |4 +- drivers/net/ethernet/emulex/benet/be_main.c | 83 ++-- drivers/net/ethernet/emulex/benet/be_roce.c | 183 +++ drivers/net/ethernet/emulex/benet/be_roce.h | 75 +++ 8 files changed, 409 insertions(+), 16 deletions(-) create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.c create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.h -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [RFC 0/2] be2net: Added functionality to support RoCE driver
-Original Message- From: Roland Dreier [mailto:rol...@purestorage.com] Sent: Thursday, March 08, 2012 12:32 AM To: Pandit, Parav; David Miller Cc: net...@vger.kernel.org; linux-rdma@vger.kernel.org Subject: Re: [RFC 0/2] be2net: Added functionality to support RoCE driver On Wed, Mar 7, 2012 at 9:47 AM, wrote: > From: Parav Pandit > > This patch is against roland's below Infiniband tree. > http://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git > Simiilar patch against netdev net-next tree will be made based on this patch. > > This patch adds functionality to support RoCE (RDMA over Ethernet) driver. > - Detecting RoCE supported adapters and creating linked list of them. > - Enabling 5 more MSIX vectors for RoCE functionality. > - Calling registered callback functions of the RoCE driver > whenever new RoCE capable device is added/removed. > - Notifying events to RoCE driver when interface is up or down. > - Provides device specific details to RoCE driver for each roce device. > - Provides low level mailbox command to be issued by RoCE driver > before it can have it own MQ. > > Parav Pandit (2): > be2net: Added function to issue mailbox cmd on MQ. > be2net: Added functionality to support RoCE driver Great, I'm really excited to be getting another upstream driver. I would suggest that when the actual RoCE driver is posted, we merge these 2 prerequisite patches + the driver through my tree, to avoid the hassle of a dependency between my tree and Dave's net-next tree. [PP] There are minor differences in be2net driver in your tree and net-next tree. So if we are fine with the interface between RoCE driver and NIC driver which is already available in this patch, then you can just merge RoCE driver patch from your tree to net-next tree. There will be separate patch against net-next tree for NIC driver which will be almost similar to this one. (Assuming these patches are OK with Dave and everyone else... they look pretty reasonable to me after a quick read) When do you expect to post the actual RoCE driver? If that's going to be a while, another possibility is to just merge these patches through net-next for 3.4, although I'm not sure about how Dave feels about merging all this stuff without seeing what will actually use it. [PP] I was just waiting for comments on this RFC/PATCH to submit separate patch for RoCE driver. But surely if RoCE driver patch helps review, I can send it right away. I'll prepare the patch for Roland's tree and send in few hours. Thanks, Roland -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 2/2] be2net: Added functionality to support RoCE driver
From: Parav Pandit
- Increased MSIX vectors by 5 for RoCE traffic.
- Added macro to check roce support on a device.
- Added device specific doorbell, msix vector fields shared with nic
functionality.
- Provides RoCE driver registration and deregistration functions.
- Added support functions which will be invoked on adapter
add/remove and port up/down events.
- Traverses through the list of adapters for invoking callback functions.
Signed-off-by: Parav Pandit
---
drivers/net/ethernet/emulex/benet/Makefile |2 +-
drivers/net/ethernet/emulex/benet/be.h | 28 -
drivers/net/ethernet/emulex/benet/be_cmds.h |1 +
drivers/net/ethernet/emulex/benet/be_hw.h |4 +-
drivers/net/ethernet/emulex/benet/be_main.c | 83 +++--
drivers/net/ethernet/emulex/benet/be_roce.c | 185 +++
drivers/net/ethernet/emulex/benet/be_roce.h | 75 +++
7 files changed, 365 insertions(+), 13 deletions(-)
create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.c
create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.h
diff --git a/drivers/net/ethernet/emulex/benet/Makefile
b/drivers/net/ethernet/emulex/benet/Makefile
index a60cd80..1a91b27 100644
--- a/drivers/net/ethernet/emulex/benet/Makefile
+++ b/drivers/net/ethernet/emulex/benet/Makefile
@@ -4,4 +4,4 @@
obj-$(CONFIG_BE2NET) += be2net.o
-be2net-y := be_main.o be_cmds.o be_ethtool.o
+be2net-y := be_main.o be_cmds.o be_ethtool.o be_roce.o
diff --git a/drivers/net/ethernet/emulex/benet/be.h
b/drivers/net/ethernet/emulex/benet/be.h
index cbdec25..767f41f 100644
--- a/drivers/net/ethernet/emulex/benet/be.h
+++ b/drivers/net/ethernet/emulex/benet/be.h
@@ -32,6 +32,7 @@
#include
#include "be_hw.h"
+#include "be_roce.h"
#define DRV_VER"4.0.100u"
#define DRV_NAME "be2net"
@@ -92,7 +93,7 @@ static inline char *nic_name(struct pci_dev *pdev)
#define MAX_RSS_QS 4 /* BE limit is 4 queues/port */
#define MAX_RX_QS (MAX_RSS_QS + 1) /* RSS qs + 1 def Rx */
#define MAX_TX_QS 8
-#define BE_MAX_MSIX_VECTORS(MAX_RX_QS + 1)/* RX + TX */
+#define BE_MAX_MSIX_VECTORS(MAX_RX_QS + 1 + 5)/* RX + TX + 5 RoCE */
#define BE_NAPI_WEIGHT 64
#define MAX_RX_POSTBE_NAPI_WEIGHT /* Frags posted at a time */
#define RX_FRAGS_REFILL_WM (RX_Q_LEN - MAX_RX_POST)
@@ -320,6 +321,7 @@ struct be_adapter {
struct msix_entry msix_entries[BE_MAX_MSIX_VECTORS];
u32 num_msix_vec;
+ u32 num_eqs;
bool isr_registered;
/* TX Rings */
@@ -372,6 +374,14 @@ struct be_adapter {
u8 transceiver;
u8 autoneg;
u8 generation; /* BladeEngine ASIC generation */
+ u32 if_type;
+ u8 __iomem *roce_db;/* Door Bell */
+ u32 roce_db_size;
+ u32 roce_db_total_size;
+ u64 roce_db_io_addr;
+ void *ocrdma_dev;
+ struct list_head entry;
+
u32 flash_status;
struct completion flash_compl;
@@ -398,6 +408,9 @@ struct be_adapter {
#define OFF0
#define lancer_chip(adapter) ((adapter->pdev->device == OC_DEVICE_ID3) || \
(adapter->pdev->device == OC_DEVICE_ID4))
+#define be_roce_supported(adapter) ((adapter->if_type == SLI_INTF_TYPE_3 || \
+ adapter->sli_family == SKYHAWK_SLI_FAMILY) && \
+ (adapter->function_mode & RDMA_ENABLED))
extern const struct ethtool_ops be_ethtool_ops;
@@ -544,4 +557,17 @@ extern void be_cq_notify(struct be_adapter *adapter, u16
qid, bool arm,
extern void be_link_status_update(struct be_adapter *adapter, u8 link_status);
extern void be_parse_stats(struct be_adapter *adapter);
extern int be_load_fw(struct be_adapter *adapter, u8 *func);
+
+/*
+ * internal function to initialize-cleanup roce device.
+ */
+extern void be_roce_dev_add(struct be_adapter *adapter);
+extern void be_roce_dev_remove(struct be_adapter *adapter);
+
+/*
+ * internal function to open-close roce device during ifup-ifdown.
+ */
+extern void be_roce_dev_open(struct be_adapter *adapter);
+extern void be_roce_dev_close(struct be_adapter *adapter);
+
#endif /* BE_H */
diff --git a/drivers/net/ethernet/emulex/benet/be_cmds.h
b/drivers/net/ethernet/emulex/benet/be_cmds.h
index dca8924..d7ad03b 100644
--- a/drivers/net/ethernet/emulex/benet/be_cmds.h
+++ b/drivers/net/ethernet/emulex/benet/be_cmds.h
@@ -1057,6 +1057,7 @@ struct be_cmd_resp_modify_eq_delay {
#define FLEX10_MODE0x400
#define VNIC_MODE 0x2
#define UMC_ENABLED0x100
+#define RDMA_ENABLED 0x4
struct be_cmd_req_query_fw_cfg {
struct be_cmd_req_hdr hdr;
u32 rsvd[31];
diff --git a/drivers/net/ethernet/emulex/benet/be_hw.h
b/drivers/net/ethernet/emulex/benet/be_hw.h
inde
[PATCH 0/2] be2net: Added functionality to support RoCE driver
From: Parav Pandit This patch adds functionality to support RoCE (RDMA over Ethernet) driver. - Detecting RoCE supported adapters and creating linked list of them. - Enabling 5 more MSIX vectors for RoCE functionality. - Calling registered callback functions of the RoCE driver whenever new RoCE capable device is added/removed. - Notifying events to RoCE driver when interface is up or down. - Provides device specific details to RoCE driver for each roce device. - Provides low level mailbox command to be issued by RoCE driver before it can have it own MQ. Parav Pandit (2): be2net: Added function to issue mailbox cmd on MQ. be2net: Added functionality to support RoCE driver drivers/net/ethernet/emulex/benet/Makefile |2 +- drivers/net/ethernet/emulex/benet/be.h | 28 - drivers/net/ethernet/emulex/benet/be_cmds.c | 39 ++ drivers/net/ethernet/emulex/benet/be_cmds.h |1 + drivers/net/ethernet/emulex/benet/be_hw.h |4 +- drivers/net/ethernet/emulex/benet/be_main.c | 83 +++-- drivers/net/ethernet/emulex/benet/be_roce.c | 185 +++ drivers/net/ethernet/emulex/benet/be_roce.h | 75 +++ 8 files changed, 404 insertions(+), 13 deletions(-) create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.c create mode 100644 drivers/net/ethernet/emulex/benet/be_roce.h -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/2] be2net: Added function to issue mailbox cmd on MQ.
From: Parav Pandit
- Added generic function to issue mailbox cmd on MQ as export function.
- RoCE driver will use this before it setups its own MQ.
Signed-off-by: Parav Pandit
---
drivers/net/ethernet/emulex/benet/be_cmds.c | 39 +++
1 files changed, 39 insertions(+), 0 deletions(-)
diff --git a/drivers/net/ethernet/emulex/benet/be_cmds.c
b/drivers/net/ethernet/emulex/benet/be_cmds.c
index 0fcb456..19037b0 100644
--- a/drivers/net/ethernet/emulex/benet/be_cmds.c
+++ b/drivers/net/ethernet/emulex/benet/be_cmds.c
@@ -15,6 +15,7 @@
* Costa Mesa, CA 92626
*/
+#include
#include "be.h"
#include "be_cmds.h"
@@ -2391,3 +2392,41 @@ err:
spin_unlock_bh(&adapter->mcc_lock);
return status;
}
+
+int be_roce_mcc_cmd(void *netdev_handle, void *wrb_payload,
+ int wrb_payload_size, u16 *cmd_status, u16 *ext_status)
+{
+ struct be_adapter *adapter = netdev_priv(netdev_handle);
+ struct be_mcc_wrb *wrb;
+ struct be_cmd_req_hdr *hdr = (struct be_cmd_req_hdr *) wrb_payload;
+ struct be_cmd_req_hdr *req;
+ struct be_cmd_resp_hdr *resp;
+ int status;
+
+ spin_lock_bh(&adapter->mcc_lock);
+
+ wrb = wrb_from_mccq(adapter);
+ if (!wrb) {
+ status = -EBUSY;
+ goto err;
+ }
+ req = embedded_payload(wrb);
+ resp = embedded_payload(wrb);
+
+ be_wrb_cmd_hdr_prepare(req, hdr->subsystem,
+ hdr->opcode, wrb_payload_size, wrb, NULL);
+ memcpy(req, wrb_payload, wrb_payload_size);
+ be_dws_cpu_to_le(req, wrb_payload_size);
+
+ status = be_mcc_notify_wait(adapter);
+ if (cmd_status)
+ *cmd_status = (status & 0x);
+ if (ext_status)
+ *ext_status = 0;
+ memcpy(wrb_payload, resp, sizeof(*resp) + resp->response_length);
+ be_dws_le_to_cpu(wrb_payload, sizeof(*resp) + resp->response_length);
+err:
+ spin_unlock_bh(&adapter->mcc_lock);
+ return status;
+}
+EXPORT_SYMBOL(be_roce_mcc_cmd);
--
1.6.0.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel
You are right, cq_lock will result into dead lock. Should there be a additional compl_handler spin_lock? I was measuring performance impact for adding it, and, irq_save() and irq_restore() variant showed additional 200 cycles, which I believe should be o.k.? Parav -Original Message- From: Steve Wise [mailto:sw...@opengridcomputing.com] Sent: Thursday, October 20, 2011 9:52 PM To: Roland Dreier Cc: Pandit, Parav; kuma...@chelsio.com; linux-rdma@vger.kernel.org; d...@chelsio.com Subject: Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel On 10/20/2011 11:06 AM, Roland Dreier wrote: > On Thu, Oct 20, 2011 at 2:11 AM, wrote: >> http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt >> >> Line no 66 to 97 states that - at a given point of time, there should be >> only one callback per CQ should be active. >> Is this ensured? >> >> compl_handler() is invoked from multiple places flush_qp() and >> post_qp_event() to my mind. > Yes, does look like an issue with this patch :( > And I think this bug exists in the T3 driver too. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel
http://lxr.linux.no/#linux+v3.0.4/Documentation/infiniband/core_locking.txt
Line no 66 to 97 states that - at a given point of time, there should be only
one callback per CQ should be active.
Is this ensured?
compl_handler() is invoked from multiple places flush_qp() and post_qp_event()
to my mind.
Parav
-Original Message-
From: linux-rdma-ow...@vger.kernel.org
[mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Steve Wise
Sent: Thursday, October 13, 2011 9:43 PM
To: Roland Dreier
Cc: Kumar Sanghvi; linux-rdma@vger.kernel.org; d...@chelsio.com
Subject: Re: [PATCH] RDMA/cxgb4: Unblock reads on comp_channel
On 10/13/2011 11:01 AM, Roland Dreier wrote:
> Would this generate a completion event even if no completion entries are
> queued?
>
I guess it can if the QP has no WRs posted at all.
> Maybe I'm misunderstanding, but this sounds like a bandaid for broken
> applications,
> and a bandaid that other hardware drivers won't implement.
>
I'm not sure other drivers have this issue. For example, when a mlx or mthca
QP moves out of RTS, does the HW flush the
pending recv WRs? For Chelsio devices, the provider driver/library must handle
this.
This logic is needed to adhere to the iwarp verbs spec which states that when
the QP moves out of RTS, all WRs that are
pending get completed with FLUSH status. For T3/T4 devices, this is all done
in software. For user mode, the provider
library has to flush the QP (IE the kernel doesn't own the queue state).
The idea is that if an application is expecting a CQ event notification when
the QP moves out of RTS, and there are only
recv wrs posted, then the T4 (and T3 does this too) driver must post this CQ
notification, in addition to marking the CQ
as "in error" which means some QP bound to this CQ needs flushing. Then when
the app wakes up and polls the CQ, the
libcxgb4 code will flush the QPs in error and thus CQEs will be inserted into
the CQ. We have seen certain applications
that rely on this event to discover that a QP has moved out of RTS. IE they
don't look at async QP events nor RDMACM
events.
Steve.
> On Thu, Oct 13, 2011 at 1:21 AM, Kumar Sanghvi wrote:
>> At the time when peer closes connection, iw_cxgb4 will not
>> send a cq event if ibqp.uobject exists. In that case, its possible
>> for user application to get blocked in ibv_get_cq_event.
>>
>> To resolve this, call the cq's comp_handler to unblock any read
>> from ibv_get_cq_event.
>>
>> Signed-off-by: Kumar Sanghvi
>> ---
>> drivers/infiniband/hw/cxgb4/qp.c |6 +-
>> 1 files changed, 5 insertions(+), 1 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/cxgb4/qp.c
>> b/drivers/infiniband/hw/cxgb4/qp.c
>> index ec3ce67..b59b56c 100644
>> --- a/drivers/infiniband/hw/cxgb4/qp.c
>> +++ b/drivers/infiniband/hw/cxgb4/qp.c
>> @@ -970,8 +970,12 @@ static void flush_qp(struct c4iw_qp *qhp)
>> if (qhp->ibqp.uobject) {
>> t4_set_wq_in_error(&qhp->wq);
>> t4_set_cq_in_error(&rchp->cq);
>> - if (schp != rchp)
>> + (*rchp->ibcq.comp_handler)(&rchp->ibcq,
>> rchp->ibcq.cq_context);
>> + if (schp != rchp) {
>> t4_set_cq_in_error(&schp->cq);
>> + (*schp->ibcq.comp_handler)(&schp->ibcq,
>> + schp->ibcq.cq_context);
>> + }
>> return;
>> }
>> __flush_qp(qhp, rchp, schp);
>> --
>> 1.7.1
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: create new library project and few queries to open source
Thanks Roland for your quick inputs. I am able to clone your github for-next branch. I'll be checkin to your branch soon. Parav -Original Message- From: Roland Dreier [mailto:rol...@purestorage.com] Sent: Thursday, October 13, 2011 11:02 PM To: Pandit, Parav Cc: linux-rdma@vger.kernel.org; v...@dev.mellanox.co.il Subject: Re: create new library project and few queries to open source On Thu, Oct 13, 2011 at 9:40 AM, wrote: > 2. For adding hardware driver for a new RDMA adapter, to which git tree(s) do > I need to submit patches? > openfabrics.org as well as kernel.org or just openfabrics is sufficient? > > git://git.openfabrics.org/ofed_1_5/linux-2.6.git, branch ofed_kernel_1_5? > and > git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git, branch > for-next? > > 4. I am little confused. > pub/scm/linux/kernel/git/roland/infiniband.git is not showing up in the list > at http://git.kernel.org. > Am I looking at wrong place? Or? > > 5. my_new_rdma_hw_driver that will be located in > drivers/infiniband/hw/my_new_rdma_hw_driver directory, does it have to > compile for all backports or only my desired kernel versions are sufficient > for OFED 1.5.x, as I don't plan to support 2.6.22 and older kernel version? Please focus on upstream kernel first, then worry about OFED backports. My for-next tree is temporarily at https://github.com/rolandd/infiniband while kernel.org recovers from the security break. But really it's fine to work against Linus's 3.1-rc as well. At this point it's probably not realistic to expect your code to get into 3.2, but it's the perfect time to start reviewing for a 3.3 merge. Looking forward to seeing another RDMA adapter supported upstream... - Roland -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
create new library project and few queries to open source
Hi, 1. I would like to submit user space library for a new RDMA adapter. Can you please do needful for creating empty project under http://git.openfabrics.org/git/projects/~ppandit/libocrdma.git? 2. For adding hardware driver for a new RDMA adapter, to which git tree(s) do I need to submit patches? openfabrics.org as well as kernel.org or just openfabrics is sufficient? git://git.openfabrics.org/ofed_1_5/linux-2.6.git, branch ofed_kernel_1_5? and git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git, branch for-next? 4. I am little confused. pub/scm/linux/kernel/git/roland/infiniband.git is not showing up in the list at http://git.kernel.org. Am I looking at wrong place? Or? 5. my_new_rdma_hw_driver that will be located in drivers/infiniband/hw/my_new_rdma_hw_driver directory, does it have to compile for all backports or only my desired kernel versions are sufficient for OFED 1.5.x, as I don't plan to support 2.6.22 and older kernel version? 6. If I have to support all backport kernel versions, do I need to submit patch for all the required kernel versions? Can this backport patches come at later stage? Regards, Parav Pandit -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
