Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Mon, May 08, 2017 at 11:55:25AM -0400, Dennis Dalessandro wrote:
> On 05/04/2017 03:42 PM, Leon Romanovsky wrote:
> > On Thu, May 04, 2017 at 03:26:13PM -0400, Dennis Dalessandro wrote:
> > > On 05/04/2017 02:45 PM, Leon Romanovsky wrote:
> > > > On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:
> > > > > On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:
> > > > > > On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:
> > > > > > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > > > > > > Following our discussion both in mailing list [1] and at the 
> > > > > > > > LPC 2016 [2],
> > > > > > > > we would like to propose this RDMA tool to be part of iproute2 
> > > > > > > > package
> > > > > > > > and finally improve this situation.
> > > > > > >
> > > > > > > Hello Leon,
> > > > > > >
> > > > > > > Although I really appreciate your work: can you clarify why you 
> > > > > > > would like to
> > > > > > > add *RDMA* functionality to an *IP routing* tool? I haven't found 
> > > > > > > any motivation
> > > > > > > for adding RDMA functionality to iproute2 in [1].
> > > > > >
> > > > > > We are planning to reuse the same infrastructure provided by 
> > > > > > iproute2,
> > > > > > like netlink parsing, access to distributions, same CLI and same 
> > > > > > standards.
> > > > > >
> > > > > > Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, 
> > > > > > HFI-VNIC.
> > > > > > Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
> > > > > > RDMA.
> > > > > >
> > > > > > I do expect that iproute2 will be installed on every machine with 
> > > > > > any
> > > > > > type of connection, including IB and OPA.
> > > > > >
> > > > > > So I think that it is enough to be part of that suite and don't 
> > > > > > invent
> > > > > > our own for one specific tool.
> > > > >
> > > > > Hello Leon,
> > > > >
> > > > > Sorry but to me that sounds like a weak argument for including RDMA 
> > > > > functionality
> > > > > in iproute2. There is already a library for communication over 
> > > > > netlink sockets,
> > > > > namely libnl. Is there functionality that is in iproute2 but not in 
> > > > > libnl and
> > > > > that is needed for the new tool? If so, have you considered to create 
> > > > > a new
> > > > > library for that functionality?
> > > >
> > > > It is not hard to create new tool, the hardest part is to ensure that 
> > > > it is
> > > > part of the distributions. Did you count how many months we are trying 
> > > > to
> > > > add rdma-core to debian?
> > >
> > > I do agree that it is a strange pairing and am not really a fan. However 
> > > at
> > > the end of the day it's just a name for a repo/package. If the iproute 
> > > folks
> > > are fine to include rdma in their repo/package, great we can leverage 
> > > their
> > > code for CLI and other common stuff.
> > >
> > > Now if the interface was something like "ip -FlagForRdma ..." I would 
> > > object
> > > to that, but the interface is "rdma ... " so from users perspective it's
> > > different tools. They don't need to care that it was sourced from a common
> > > git repo.
> > >
> > > Just as an aside this already works a bit with OPA:
> > >
> > >  $ ./rdma link
> > > 1/1: hfi1_0/1: ifname NONE cap_mask 0x00410022 lid 0x1 lid_mask_count 0
> > > link_layer InfiniBand
> > >  phys_state 5: LinkUp rate 100 Gb/sec (4X EDR) sm_lid 0x1 sm_sl 0
> > > state 4: ACTIVE
> > >
> > > Leon I'll get you more feedback and testing, I've just been really bogged
> > > down this week, sorry.
> >
> > Thanks Denny,
> >
> > Before you are starting to test it, can you please provide your feedback
> > on my initial questions? Usability and need of sysfs.
> > 
> > This is initial phase to understand if user experience for this tool fits
> > RDMA and netdev communities exepectations. Also I would like to get feedback
> > if it is really worth to provide legacy sysfs for old kernels, or maybe I 
> > should
> > implement netlink from the beginning and abandon sysfs completely.
> > -
>
> For the initial phase I think you did the right thing by reading sysfs. I
> like the ability for the tool to be compatible with legacy kernels, but at
> the same time I don't know if it's worth the hassle. I won't fight too hard
> either way.
>
> Perhaps we should take a stab and seeing what a dual sysfs/netlink interface
> would look like, and see just how hard and complicated it really is. Then we
> can make that call. You already have the sysfs version, and have to do
> netink anyway, so let's not rip out what's there just yet, this is an RFC
> after all, not like you are asking for this exact version to be merged yet.

Not at all, it is too early to merge it, at the end it is POC.

>
> As far as usability, I think what's here is a great start and we can
> continue to refine. I'm particularly interested in the stats capabilities.
> In particular being able to filter what stats are shown and watch as 

Re: [RFC iproute2 0/8] RDMA tool

[Andrew Lunn]

> Several companies maintain embedded Linux
> distributions and tools to build software images. These tools provide a user
> interface that allows to select what packages go into such an image.

The tools allow you to select what binary packages are placed into the
image. You can build multiple binary packages from one source package.
Desktop distributions are not likely to do this for something as small
as iproute2. But embedded distributions can easily break up iproute2
into a number of smaller packages, as you suggested, tipc, devlink,
tc, bridge, ss, etc.

Openwrt does exactly this:

https://github.com/openwrt-mirror/openwrt/blob/master/package/network/utils/iproute2/Makefile

Andrew

Re: [RFC iproute2 0/8] RDMA tool

[Dennis Dalessandro]

On 05/04/2017 03:42 PM, Leon Romanovsky wrote:

On Thu, May 04, 2017 at 03:26:13PM -0400, Dennis Dalessandro wrote:

On 05/04/2017 02:45 PM, Leon Romanovsky wrote:

On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:

On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:

On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.


Hello Leon,

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].


We are planning to reuse the same infrastructure provided by iproute2,
like netlink parsing, access to distributions, same CLI and same standards.

Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, HFI-VNIC.
Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
RDMA.

I do expect that iproute2 will be installed on every machine with any
type of connection, including IB and OPA.

So I think that it is enough to be part of that suite and don't invent
our own for one specific tool.


Hello Leon,

Sorry but to me that sounds like a weak argument for including RDMA 
functionality
in iproute2. There is already a library for communication over netlink sockets,
namely libnl. Is there functionality that is in iproute2 but not in libnl and
that is needed for the new tool? If so, have you considered to create a new
library for that functionality?


It is not hard to create new tool, the hardest part is to ensure that it is
part of the distributions. Did you count how many months we are trying to
add rdma-core to debian?


I do agree that it is a strange pairing and am not really a fan. However at
the end of the day it's just a name for a repo/package. If the iproute folks
are fine to include rdma in their repo/package, great we can leverage their
code for CLI and other common stuff.

Now if the interface was something like "ip -FlagForRdma ..." I would object
to that, but the interface is "rdma ... " so from users perspective it's
different tools. They don't need to care that it was sourced from a common
git repo.

Just as an aside this already works a bit with OPA:

 $ ./rdma link
1/1: hfi1_0/1: ifname NONE cap_mask 0x00410022 lid 0x1 lid_mask_count 0
link_layer InfiniBand
 phys_state 5: LinkUp rate 100 Gb/sec (4X EDR) sm_lid 0x1 sm_sl 0
state 4: ACTIVE

Leon I'll get you more feedback and testing, I've just been really bogged
down this week, sorry.


Thanks Denny,

Before you are starting to test it, can you please provide your feedback
on my initial questions? Usability and need of sysfs.

This is initial phase to understand if user experience for this tool fits
RDMA and netdev communities exepectations. Also I would like to get feedback
if it is really worth to provide legacy sysfs for old kernels, or maybe I should
implement netlink from the beginning and abandon sysfs completely.
-


For the initial phase I think you did the right thing by reading sysfs. 
I like the ability for the tool to be compatible with legacy kernels, 
but at the same time I don't know if it's worth the hassle. I won't 
fight too hard either way.


Perhaps we should take a stab and seeing what a dual sysfs/netlink 
interface would look like, and see just how hard and complicated it 
really is. Then we can make that call. You already have the sysfs 
version, and have to do netink anyway, so let's not rip out what's there 
just yet, this is an RFC after all, not like you are asking for this 
exact version to be merged yet.


As far as usability, I think what's here is a great start and we can 
continue to refine. I'm particularly interested in the stats 
capabilities. In particular being able to filter what stats are shown 
and watch as they change over time. RDMA devices have lots of counters 
and stats. A common tool for users to be able to get that data across HW 
types would be a really good thing.


-Denny

Re: [RFC iproute2 0/8] RDMA tool

[Stephen Hemminger]

On Mon, 8 May 2017 15:19:28 +
Bart Van Assche  wrote:

> On Sun, 2017-05-07 at 12:20 +0200, Jiri Pirko wrote:
> > Sat, May 06, 2017 at 04:40:24PM CEST, bart.vanass...@sandisk.com wrote:  
> > > On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:  
> > > > Thu, May 04, 2017 at 08:10:54PM CEST, bart.vanass...@sandisk.com wrote: 
> > > >  
> > > > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:  
> > > > > > Following our discussion both in mailing list [1] and at the LPC 
> > > > > > 2016 [2],
> > > > > > we would like to propose this RDMA tool to be part of iproute2 
> > > > > > package
> > > > > > and finally improve this situation.  
> > > > > 
> > > > > Although I really appreciate your work: can you clarify why you would 
> > > > > like to
> > > > > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > > > > motivation
> > > > > for adding RDMA functionality to iproute2 in [1].  
> > > > 
> > > > Bart, please realize that iproute2 is much more than "*IP routing* 
> > > > tool".
> > > > I understand you got confused by the name. Please see sources. Your 
> > > > comment
> > > > is totally pointless...  
> > > 
> > > I asked for a clarification that should have been in the cover letter but 
> > > that
> > > was missing from that cover letter. So I think that was the right thing 
> > > to do  
> > 
> > I think that was just complete misunderstanding about what iproute2 is.  
> 
> Hello Jiri,
> 
> I do not agree with your reply. The abbreviation "IP" occurs in the package
> name and that is a reference to the "Internet Protocol". As far as I know
> originally the iproute2 package contained only tools related to the Internet
> Protocol. Other tools, e.g. the tipc tool, were added later on. What I'm
> wondering about is whether it really is a good idea to add tools like tipc
> and rdma to the iproute2 package. The iproute2 package is so essential that
> it gets installed on every Linux system, including embedded systems and
> smartphones based on Linux. Several companies maintain embedded Linux
> distributions and tools to build software images. These tools provide a user
> interface that allows to select what packages go into such an image. Adding
> tools like tipc and rdma to the iproute2 package makes it harder than
> necessary for those who build software images for embedded devices to
> minimize the size of such an image. As you probably know even today the size
> of a software image still matters for embedded devices. Something else I have
> been wondering about is whether bundling the tipc and rdma tools in the
> iproute2 package will make the job harder of people who build Android ROMs?
> The ip tool is present in every Android ROM, and the size of these ROMs 
> matters
> because the larger these ROMs are the less space remains for apps.
> 
> Bart.

I would assume embedded world does not use the standard distro model of "got to
have them all". The sane way to do builds for embedded is build everything
in the source, but selectively install components based on a Bill Of Materials
file.

Re: [RFC iproute2 0/8] RDMA tool

[Bart Van Assche]

On Sun, 2017-05-07 at 12:20 +0200, Jiri Pirko wrote:
> Sat, May 06, 2017 at 04:40:24PM CEST, bart.vanass...@sandisk.com wrote:
> > On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:
> > > Thu, May 04, 2017 at 08:10:54PM CEST, bart.vanass...@sandisk.com wrote:
> > > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > > > Following our discussion both in mailing list [1] and at the LPC 2016 
> > > > > [2],
> > > > > we would like to propose this RDMA tool to be part of iproute2 package
> > > > > and finally improve this situation.
> > > > 
> > > > Although I really appreciate your work: can you clarify why you would 
> > > > like to
> > > > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > > > motivation
> > > > for adding RDMA functionality to iproute2 in [1].
> > > 
> > > Bart, please realize that iproute2 is much more than "*IP routing* tool".
> > > I understand you got confused by the name. Please see sources. Your 
> > > comment
> > > is totally pointless...
> > 
> > I asked for a clarification that should have been in the cover letter but 
> > that
> > was missing from that cover letter. So I think that was the right thing to 
> > do
> 
> I think that was just complete misunderstanding about what iproute2 is.

Hello Jiri,

I do not agree with your reply. The abbreviation "IP" occurs in the package
name and that is a reference to the "Internet Protocol". As far as I know
originally the iproute2 package contained only tools related to the Internet
Protocol. Other tools, e.g. the tipc tool, were added later on. What I'm
wondering about is whether it really is a good idea to add tools like tipc
and rdma to the iproute2 package. The iproute2 package is so essential that
it gets installed on every Linux system, including embedded systems and
smartphones based on Linux. Several companies maintain embedded Linux
distributions and tools to build software images. These tools provide a user
interface that allows to select what packages go into such an image. Adding
tools like tipc and rdma to the iproute2 package makes it harder than
necessary for those who build software images for embedded devices to
minimize the size of such an image. As you probably know even today the size
of a software image still matters for embedded devices. Something else I have
been wondering about is whether bundling the tipc and rdma tools in the
iproute2 package will make the job harder of people who build Android ROMs?
The ip tool is present in every Android ROM, and the size of these ROMs matters
because the larger these ROMs are the less space remains for apps.

Bart.

Re: [RFC iproute2 0/8] RDMA tool

[Knut Omang]

On Sun, 2017-05-07 at 09:33 +0300, Leon Romanovsky wrote:
> On Sat, May 06, 2017 at 12:48:26PM +0200, Jiri Pirko wrote:
> > Fri, May 05, 2017 at 03:17:54PM CEST, l...@kernel.org wrote:
> > >On Fri, May 05, 2017 at 08:54:57AM +0200, Jiri Benc wrote:
> > >> On Thu,  4 May 2017 21:02:08 +0300, Leon Romanovsky wrote:
> > >> > In order to close object model, ensure reuse of existing code and make 
> > >> > this
> > >> > tool usable from day one, we decided to implement wrappers over legacy 
> > >> > sysfs
> > >> > prior to implementing netlink functionality. As a nice bonus, it will 
> > >> > allow
> > >> > to use this tool with old kernels too.
> > >>
> > >> This sounds wrong. We don't support legacy ioctl interface for the 'ip'
> > >> command, either. I think rdma should be converted to netlink first and
> > >> the new tool should only use netlink.
> > >
> > >RDMA in slightly different situation than "ip" tool was. "ip" was 
> > >implemented
> > >when tools like ifconfig existed. It allowed to old and new systems to be
> > >configured to some degree. In RDMA community, there are no similar tools 
> > >like
> > >"ifconfig". Implementation in netlink-only interface will leave old 
> > >systems without
> > >common tool at all.
> > >
> > >As an upstream-oriented person, I personally fine with that, but anyway 
> > >would
> > >like to get wider agreement/disagreement on that, before removing sysfs
> > >parsing logic from the rdmatool.
> >
> > I tend to agree with Jiri Benc. I fear that supporting sysfs + netlink
> > api later on for the same things will make the code unnecessary complex.
> > Also, the legacy sysfs will most likely stay there forever so there will
> > be no actual motivation to port the existing things to the new netlink
> > api.
> >
> > For the prototyping purposes, I belive that what you did makes perfect
> > sense. But for the actual mergable version, my feeling is that we need
> > to strictly stick with new netlink rdma interface and just forget about
> > the old sysfs one. Distros would have to backport the new kernel
> > rdma netlink api.
> 
> Thanks,
> It looks like that most of the comments are in favor of netlink-only
> solution.

Leon, I like the thought bw comp support. After all this is a user level tool 
so it should
be possible to make a clean implementation that makes the old stuff easy to 
remove 
at some point. It will also attract users much sooner than if they have to have 
their own if-then-else logic around everything to be able to support old and 
new.

> > Yes, this will be little bit more painful at the beginning, but in the
> > long run, I believe it will save some severe headaches.
> >

IMHO, some headache will be there anyway, just a matter of how how far out it 
gets.

Knut

Re: [RFC iproute2 0/8] RDMA tool

[Stephen Hemminger]

On Sun, 7 May 2017 09:33:29 +0300
Leon Romanovsky  wrote:

> On Sat, May 06, 2017 at 12:48:26PM +0200, Jiri Pirko wrote:
> > Fri, May 05, 2017 at 03:17:54PM CEST, l...@kernel.org wrote:  
> > >On Fri, May 05, 2017 at 08:54:57AM +0200, Jiri Benc wrote:  
> > >> On Thu,  4 May 2017 21:02:08 +0300, Leon Romanovsky wrote:  
> > >> > In order to close object model, ensure reuse of existing code and make 
> > >> > this
> > >> > tool usable from day one, we decided to implement wrappers over legacy 
> > >> > sysfs
> > >> > prior to implementing netlink functionality. As a nice bonus, it will 
> > >> > allow
> > >> > to use this tool with old kernels too.  
> > >>
> > >> This sounds wrong. We don't support legacy ioctl interface for the 'ip'
> > >> command, either. I think rdma should be converted to netlink first and
> > >> the new tool should only use netlink.  
> > >
> > >RDMA in slightly different situation than "ip" tool was. "ip" was 
> > >implemented
> > >when tools like ifconfig existed. It allowed to old and new systems to be
> > >configured to some degree. In RDMA community, there are no similar tools 
> > >like
> > >"ifconfig". Implementation in netlink-only interface will leave old 
> > >systems without
> > >common tool at all.
> > >
> > >As an upstream-oriented person, I personally fine with that, but anyway 
> > >would
> > >like to get wider agreement/disagreement on that, before removing sysfs
> > >parsing logic from the rdmatool.  
> >
> > I tend to agree with Jiri Benc. I fear that supporting sysfs + netlink
> > api later on for the same things will make the code unnecessary complex.
> > Also, the legacy sysfs will most likely stay there forever so there will
> > be no actual motivation to port the existing things to the new netlink
> > api.
> >
> > For the prototyping purposes, I belive that what you did makes perfect
> > sense. But for the actual mergable version, my feeling is that we need
> > to strictly stick with new netlink rdma interface and just forget about
> > the old sysfs one. Distros would have to backport the new kernel
> > rdma netlink api.  
> 
> Thanks,
> It looks like that most of the comments are in favor of netlink-only
> solution.

If current (like 4.10 or later) kernel support netlink only solution, that 
makes sense.
When I created bridge command; it also abandoned the old ioctl interface.


pgpOZX3v0e7qH.pgp
Description: OpenPGP digital signature

Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Sat, May 06, 2017 at 12:48:26PM +0200, Jiri Pirko wrote:
> Fri, May 05, 2017 at 03:17:54PM CEST, l...@kernel.org wrote:
> >On Fri, May 05, 2017 at 08:54:57AM +0200, Jiri Benc wrote:
> >> On Thu,  4 May 2017 21:02:08 +0300, Leon Romanovsky wrote:
> >> > In order to close object model, ensure reuse of existing code and make 
> >> > this
> >> > tool usable from day one, we decided to implement wrappers over legacy 
> >> > sysfs
> >> > prior to implementing netlink functionality. As a nice bonus, it will 
> >> > allow
> >> > to use this tool with old kernels too.
> >>
> >> This sounds wrong. We don't support legacy ioctl interface for the 'ip'
> >> command, either. I think rdma should be converted to netlink first and
> >> the new tool should only use netlink.
> >
> >RDMA in slightly different situation than "ip" tool was. "ip" was implemented
> >when tools like ifconfig existed. It allowed to old and new systems to be
> >configured to some degree. In RDMA community, there are no similar tools like
> >"ifconfig". Implementation in netlink-only interface will leave old systems 
> >without
> >common tool at all.
> >
> >As an upstream-oriented person, I personally fine with that, but anyway would
> >like to get wider agreement/disagreement on that, before removing sysfs
> >parsing logic from the rdmatool.
>
> I tend to agree with Jiri Benc. I fear that supporting sysfs + netlink
> api later on for the same things will make the code unnecessary complex.
> Also, the legacy sysfs will most likely stay there forever so there will
> be no actual motivation to port the existing things to the new netlink
> api.
>
> For the prototyping purposes, I belive that what you did makes perfect
> sense. But for the actual mergable version, my feeling is that we need
> to strictly stick with new netlink rdma interface and just forget about
> the old sysfs one. Distros would have to backport the new kernel
> rdma netlink api.

Thanks,
It looks like that most of the comments are in favor of netlink-only
solution.

>
> Yes, this will be little bit more painful at the beginning, but in the
> long run, I believe it will save some severe headaches.
>


signature.asc
Description: PGP signature

Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Sat, May 06, 2017 at 02:40:24PM +, Bart Van Assche wrote:
> On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:
> > Thu, May 04, 2017 at 08:10:54PM CEST, bart.vanass...@sandisk.com wrote:
> > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > > Following our discussion both in mailing list [1] and at the LPC 2016 
> > > > [2],
> > > > we would like to propose this RDMA tool to be part of iproute2 package
> > > > and finally improve this situation.
> > >
> > > Although I really appreciate your work: can you clarify why you would 
> > > like to
> > > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > > motivation
> > > for adding RDMA functionality to iproute2 in [1].
> >
> > Bart, please realize that iproute2 is much more than "*IP routing* tool".
> > I understand you got confused by the name. Please see sources. Your comment
> > is totally pointless...
>
> I asked for a clarification that should have been in the cover letter but that
> was missing from that cover letter. So I think that was the right thing to do
> instead of pointless. BTW, can you explain why you are using an e-mail address
> that is hiding that you are a Mellanox employee?

For the same reason as I do. It is much easier to use outside servers
than Mellanox's IT infrastructure.

Right now, I'm speaking for myself, but for me, to properly answer with 
@mellanox.com,
I need to use very specific mail setup, while for any other addresses I can and 
use any
sane setup, including mobile application.

>
> Bart.


signature.asc
Description: PGP signature

Re: [RFC iproute2 0/8] RDMA tool

[Jiri Pirko]

Sat, May 06, 2017 at 04:40:24PM CEST, bart.vanass...@sandisk.com wrote:
>On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:
>> Thu, May 04, 2017 at 08:10:54PM CEST, bart.vanass...@sandisk.com wrote:
>> > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
>> > > Following our discussion both in mailing list [1] and at the LPC 2016 
>> > > [2],
>> > > we would like to propose this RDMA tool to be part of iproute2 package
>> > > and finally improve this situation.
>> > 
>> > Although I really appreciate your work: can you clarify why you would like 
>> > to
>> > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
>> > motivation
>> > for adding RDMA functionality to iproute2 in [1].
>> 
>> Bart, please realize that iproute2 is much more than "*IP routing* tool".
>> I understand you got confused by the name. Please see sources. Your comment
>> is totally pointless...
>
>I asked for a clarification that should have been in the cover letter but that
>was missing from that cover letter. So I think that was the right thing to do

I think that was just complete misunderstanding about what iproute2 is.


>instead of pointless. BTW, can you explain why you are using an e-mail address
>that is hiding that you are a Mellanox employee?

Who sais I have to do that? This is funny...

Re: [RFC iproute2 0/8] RDMA tool

[Bart Van Assche]

On Sat, 2017-05-06 at 12:40 +0200, Jiri Pirko wrote:
> Thu, May 04, 2017 at 08:10:54PM CEST, bart.vanass...@sandisk.com wrote:
> > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > Following our discussion both in mailing list [1] and at the LPC 2016 [2],
> > > we would like to propose this RDMA tool to be part of iproute2 package
> > > and finally improve this situation.
> > 
> > Although I really appreciate your work: can you clarify why you would like 
> > to
> > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > motivation
> > for adding RDMA functionality to iproute2 in [1].
> 
> Bart, please realize that iproute2 is much more than "*IP routing* tool".
> I understand you got confused by the name. Please see sources. Your comment
> is totally pointless...

I asked for a clarification that should have been in the cover letter but that
was missing from that cover letter. So I think that was the right thing to do
instead of pointless. BTW, can you explain why you are using an e-mail address
that is hiding that you are a Mellanox employee?

Bart.

Re: [RFC iproute2 0/8] RDMA tool

[Jiri Pirko]

Fri, May 05, 2017 at 03:17:54PM CEST, l...@kernel.org wrote:
>On Fri, May 05, 2017 at 08:54:57AM +0200, Jiri Benc wrote:
>> On Thu,  4 May 2017 21:02:08 +0300, Leon Romanovsky wrote:
>> > In order to close object model, ensure reuse of existing code and make this
>> > tool usable from day one, we decided to implement wrappers over legacy 
>> > sysfs
>> > prior to implementing netlink functionality. As a nice bonus, it will allow
>> > to use this tool with old kernels too.
>>
>> This sounds wrong. We don't support legacy ioctl interface for the 'ip'
>> command, either. I think rdma should be converted to netlink first and
>> the new tool should only use netlink.
>
>RDMA in slightly different situation than "ip" tool was. "ip" was implemented
>when tools like ifconfig existed. It allowed to old and new systems to be
>configured to some degree. In RDMA community, there are no similar tools like
>"ifconfig". Implementation in netlink-only interface will leave old systems 
>without
>common tool at all.
>
>As an upstream-oriented person, I personally fine with that, but anyway would
>like to get wider agreement/disagreement on that, before removing sysfs
>parsing logic from the rdmatool.

I tend to agree with Jiri Benc. I fear that supporting sysfs + netlink
api later on for the same things will make the code unnecessary complex.
Also, the legacy sysfs will most likely stay there forever so there will
be no actual motivation to port the existing things to the new netlink
api.

For the prototyping purposes, I belive that what you did makes perfect
sense. But for the actual mergable version, my feeling is that we need
to strictly stick with new netlink rdma interface and just forget about
the old sysfs one. Distros would have to backport the new kernel
rdma netlink api.

Yes, this will be little bit more painful at the beginning, but in the
long run, I believe it will save some severe headaches.

Re: [RFC iproute2 0/8] RDMA tool

[Jiri Pirko]

Thu, May 04, 2017 at 08:10:54PM CEST, bart.vanass...@sandisk.com wrote:
>On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
>> Following our discussion both in mailing list [1] and at the LPC 2016 [2],
>> we would like to propose this RDMA tool to be part of iproute2 package
>> and finally improve this situation.
>
>Hello Leon,
>
>Although I really appreciate your work: can you clarify why you would like to
>add *RDMA* functionality to an *IP routing* tool? I haven't found any 
>motivation
>for adding RDMA functionality to iproute2 in [1].

Bart, please realize that iproute2 is much more than "*IP routing* tool".
I understand you got confused by the name. Please see sources. Your comment
is totally pointless...

Re: [RFC iproute2 0/8] RDMA tool

[Bart Van Assche]

On Thu, 2017-05-04 at 21:45 +0300, Leon Romanovsky wrote:
> It is not hard to create new tool, the hardest part is to ensure that it is
> part of the distributions. Did you count how many months we are trying to
> add rdma-core to debian?

Hello Leon,

Sorry but I was not aware that the effort to add rdma-core to Debian was taking
that long. Please let me know if I can help with that effort.

Bart.

Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Fri, May 05, 2017 at 08:54:57AM +0200, Jiri Benc wrote:
> On Thu,  4 May 2017 21:02:08 +0300, Leon Romanovsky wrote:
> > In order to close object model, ensure reuse of existing code and make this
> > tool usable from day one, we decided to implement wrappers over legacy sysfs
> > prior to implementing netlink functionality. As a nice bonus, it will allow
> > to use this tool with old kernels too.
>
> This sounds wrong. We don't support legacy ioctl interface for the 'ip'
> command, either. I think rdma should be converted to netlink first and
> the new tool should only use netlink.

RDMA in slightly different situation than "ip" tool was. "ip" was implemented
when tools like ifconfig existed. It allowed to old and new systems to be
configured to some degree. In RDMA community, there are no similar tools like
"ifconfig". Implementation in netlink-only interface will leave old systems 
without
common tool at all.

As an upstream-oriented person, I personally fine with that, but anyway would
like to get wider agreement/disagreement on that, before removing sysfs
parsing logic from the rdmatool.

Thanks

>
>  Jiri


signature.asc
Description: PGP signature

Re: [RFC iproute2 0/8] RDMA tool

[Jiri Benc]

On Thu,  4 May 2017 21:02:08 +0300, Leon Romanovsky wrote:
> In order to close object model, ensure reuse of existing code and make this
> tool usable from day one, we decided to implement wrappers over legacy sysfs
> prior to implementing netlink functionality. As a nice bonus, it will allow
> to use this tool with old kernels too.

This sounds wrong. We don't support legacy ioctl interface for the 'ip'
command, either. I think rdma should be converted to netlink first and
the new tool should only use netlink.

 Jiri

Re: [RFC iproute2 0/8] RDMA tool

[Stephen Hemminger]

On Thu, 04 May 2017 16:45:58 -0400
Doug Ledford  wrote:

> On Thu, 2017-05-04 at 15:26 -0400, Dennis Dalessandro wrote:
> > On 05/04/2017 02:45 PM, Leon Romanovsky wrote:  
> > > 
> > > On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:  
> > > > 
> > > > On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:  
> > > > > 
> > > > > On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche
> > > > > wrote:  
> > > > > > 
> > > > > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:  
> > > > > > > 
> > > > > > > Following our discussion both in mailing list [1] and at
> > > > > > > the LPC 2016 [2],
> > > > > > > we would like to propose this RDMA tool to be part of
> > > > > > > iproute2 package
> > > > > > > and finally improve this situation.  
> > > > > > 
> > > > > > Hello Leon,
> > > > > > 
> > > > > > Although I really appreciate your work: can you clarify why
> > > > > > you would like to
> > > > > > add *RDMA* functionality to an *IP routing* tool? I haven't
> > > > > > found any motivation
> > > > > > for adding RDMA functionality to iproute2 in [1].  
> > > > > 
> > > > > We are planning to reuse the same infrastructure provided by
> > > > > iproute2,
> > > > > like netlink parsing, access to distributions, same CLI and
> > > > > same standards.
> > > > > 
> > > > > Right now, RDMA is already tightened to netdev: iWARP, RoCE,
> > > > > IPoIB, HFI-VNIC.
> > > > > Many drivers (mlx, qed, i40, cxgb) are sharing code between net
> > > > > and
> > > > > RDMA.
> > > > > 
> > > > > I do expect that iproute2 will be installed on every machine
> > > > > with any
> > > > > type of connection, including IB and OPA.
> > > > > 
> > > > > So I think that it is enough to be part of that suite and don't
> > > > > invent
> > > > > our own for one specific tool.  
> > > > 
> > > > Hello Leon,
> > > > 
> > > > Sorry but to me that sounds like a weak argument for including
> > > > RDMA functionality
> > > > in iproute2. There is already a library for communication over
> > > > netlink sockets,
> > > > namely libnl. Is there functionality that is in iproute2 but not
> > > > in libnl and
> > > > that is needed for the new tool? If so, have you considered to
> > > > create a new
> > > > library for that functionality?  
> > > 
> > > It is not hard to create new tool, the hardest part is to ensure
> > > that it is
> > > part of the distributions. Did you count how many months we are
> > > trying to
> > > add rdma-core to debian?  
> > 
> > I do agree that it is a strange pairing and am not really a fan.
> > However 
> > at the end of the day it's just a name for a repo/package. If the 
> > iproute folks are fine to include rdma in their repo/package, great
> > we 
> > can leverage their code for CLI and other common stuff.  
> 
> If you look into the iproute2 package, it becomes clear that the name
> iproute2 is historical and not really accurate any more.  It contains
> things like the bridge control software, tc for controlling send
> queues, and many things network related but not routing related.  The
> rdma tool is a perfectly fine fit in the sense that it is an additional
> network management tool IMO.
> 
> For reference, here's the list of stuff already in iproute on my Fedora
> 24 box:
> 
> /usr/sbin/arpd
> /usr/sbin/bridge
> /usr/sbin/cbq
> /usr/sbin/ctstat
> /usr/sbin/genl
> /usr/sbin/ifcfg
> /usr/sbin/ifstat
> /usr/sbin/ip
> /usr/sbin/lnstat
> /usr/sbin/nstat
> /usr/sbin/routef
> /usr/sbin/routel
> /usr/sbin/rtacct
> /usr/sbin/rtmon
> /usr/sbin/rtpr
> /usr/sbin/rtstat
> /usr/sbin/ss
> /usr/sbin/tc
> /usr/sbin/tipc
> 
> And in fact, if you check, tipc is almost similar to RDMA ;-)  So, I
> suggest people not get hung up on the name iproute2, the fit is fine
> when you look deeper into the nature of the package.
> 

Iproute2 is a collection like busybox. It has bridging, devlink and tipc 
already.

Re: [RFC iproute2 0/8] RDMA tool

[Doug Ledford]

On Thu, 2017-05-04 at 15:26 -0400, Dennis Dalessandro wrote:
> On 05/04/2017 02:45 PM, Leon Romanovsky wrote:
> > 
> > On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:
> > > 
> > > On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:
> > > > 
> > > > On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche
> > > > wrote:
> > > > > 
> > > > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > > > > 
> > > > > > Following our discussion both in mailing list [1] and at
> > > > > > the LPC 2016 [2],
> > > > > > we would like to propose this RDMA tool to be part of
> > > > > > iproute2 package
> > > > > > and finally improve this situation.
> > > > > 
> > > > > Hello Leon,
> > > > > 
> > > > > Although I really appreciate your work: can you clarify why
> > > > > you would like to
> > > > > add *RDMA* functionality to an *IP routing* tool? I haven't
> > > > > found any motivation
> > > > > for adding RDMA functionality to iproute2 in [1].
> > > > 
> > > > We are planning to reuse the same infrastructure provided by
> > > > iproute2,
> > > > like netlink parsing, access to distributions, same CLI and
> > > > same standards.
> > > > 
> > > > Right now, RDMA is already tightened to netdev: iWARP, RoCE,
> > > > IPoIB, HFI-VNIC.
> > > > Many drivers (mlx, qed, i40, cxgb) are sharing code between net
> > > > and
> > > > RDMA.
> > > > 
> > > > I do expect that iproute2 will be installed on every machine
> > > > with any
> > > > type of connection, including IB and OPA.
> > > > 
> > > > So I think that it is enough to be part of that suite and don't
> > > > invent
> > > > our own for one specific tool.
> > > 
> > > Hello Leon,
> > > 
> > > Sorry but to me that sounds like a weak argument for including
> > > RDMA functionality
> > > in iproute2. There is already a library for communication over
> > > netlink sockets,
> > > namely libnl. Is there functionality that is in iproute2 but not
> > > in libnl and
> > > that is needed for the new tool? If so, have you considered to
> > > create a new
> > > library for that functionality?
> > 
> > It is not hard to create new tool, the hardest part is to ensure
> > that it is
> > part of the distributions. Did you count how many months we are
> > trying to
> > add rdma-core to debian?
> 
> I do agree that it is a strange pairing and am not really a fan.
> However 
> at the end of the day it's just a name for a repo/package. If the 
> iproute folks are fine to include rdma in their repo/package, great
> we 
> can leverage their code for CLI and other common stuff.

If you look into the iproute2 package, it becomes clear that the name
iproute2 is historical and not really accurate any more.  It contains
things like the bridge control software, tc for controlling send
queues, and many things network related but not routing related.  The
rdma tool is a perfectly fine fit in the sense that it is an additional
network management tool IMO.

For reference, here's the list of stuff already in iproute on my Fedora
24 box:

/usr/sbin/arpd
/usr/sbin/bridge
/usr/sbin/cbq
/usr/sbin/ctstat
/usr/sbin/genl
/usr/sbin/ifcfg
/usr/sbin/ifstat
/usr/sbin/ip
/usr/sbin/lnstat
/usr/sbin/nstat
/usr/sbin/routef
/usr/sbin/routel
/usr/sbin/rtacct
/usr/sbin/rtmon
/usr/sbin/rtpr
/usr/sbin/rtstat
/usr/sbin/ss
/usr/sbin/tc
/usr/sbin/tipc

And in fact, if you check, tipc is almost similar to RDMA ;-)  So, I
suggest people not get hung up on the name iproute2, the fit is fine
when you look deeper into the nature of the package.

-- 
Doug Ledford 
    GPG KeyID: B826A3330E572FDD
   
Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Thu, May 04, 2017 at 03:26:13PM -0400, Dennis Dalessandro wrote:
> On 05/04/2017 02:45 PM, Leon Romanovsky wrote:
> > On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:
> > > On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:
> > > > On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:
> > > > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > > > > Following our discussion both in mailing list [1] and at the LPC 
> > > > > > 2016 [2],
> > > > > > we would like to propose this RDMA tool to be part of iproute2 
> > > > > > package
> > > > > > and finally improve this situation.
> > > > >
> > > > > Hello Leon,
> > > > >
> > > > > Although I really appreciate your work: can you clarify why you would 
> > > > > like to
> > > > > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > > > > motivation
> > > > > for adding RDMA functionality to iproute2 in [1].
> > > >
> > > > We are planning to reuse the same infrastructure provided by iproute2,
> > > > like netlink parsing, access to distributions, same CLI and same 
> > > > standards.
> > > >
> > > > Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, 
> > > > HFI-VNIC.
> > > > Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
> > > > RDMA.
> > > >
> > > > I do expect that iproute2 will be installed on every machine with any
> > > > type of connection, including IB and OPA.
> > > >
> > > > So I think that it is enough to be part of that suite and don't invent
> > > > our own for one specific tool.
> > >
> > > Hello Leon,
> > >
> > > Sorry but to me that sounds like a weak argument for including RDMA 
> > > functionality
> > > in iproute2. There is already a library for communication over netlink 
> > > sockets,
> > > namely libnl. Is there functionality that is in iproute2 but not in libnl 
> > > and
> > > that is needed for the new tool? If so, have you considered to create a 
> > > new
> > > library for that functionality?
> >
> > It is not hard to create new tool, the hardest part is to ensure that it is
> > part of the distributions. Did you count how many months we are trying to
> > add rdma-core to debian?
>
> I do agree that it is a strange pairing and am not really a fan. However at
> the end of the day it's just a name for a repo/package. If the iproute folks
> are fine to include rdma in their repo/package, great we can leverage their
> code for CLI and other common stuff.
>
> Now if the interface was something like "ip -FlagForRdma ..." I would object
> to that, but the interface is "rdma ... " so from users perspective it's
> different tools. They don't need to care that it was sourced from a common
> git repo.
>
> Just as an aside this already works a bit with OPA:
>
>  $ ./rdma link
> 1/1: hfi1_0/1: ifname NONE cap_mask 0x00410022 lid 0x1 lid_mask_count 0
> link_layer InfiniBand
>  phys_state 5: LinkUp rate 100 Gb/sec (4X EDR) sm_lid 0x1 sm_sl 0
> state 4: ACTIVE
>
> Leon I'll get you more feedback and testing, I've just been really bogged
> down this week, sorry.

Thanks Denny,

Before you are starting to test it, can you please provide your feedback
on my initial questions? Usability and need of sysfs.


This is initial phase to understand if user experience for this tool fits
RDMA and netdev communities exepectations. Also I would like to get feedback
if it is really worth to provide legacy sysfs for old kernels, or maybe I should
implement netlink from the beginning and abandon sysfs completely.
-

P.S. I believe this will give you wrong output, because it parses IB port 
cap_mask.
$./rdma link show hfi1_0/1 cap_mask

Thanks

>
> -Denny
>
>
>


signature.asc
Description: PGP signature

Re: [RFC iproute2 0/8] RDMA tool

[Dennis Dalessandro]

On 05/04/2017 02:45 PM, Leon Romanovsky wrote:

On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:

On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:

On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.


Hello Leon,

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].


We are planning to reuse the same infrastructure provided by iproute2,
like netlink parsing, access to distributions, same CLI and same standards.

Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, HFI-VNIC.
Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
RDMA.

I do expect that iproute2 will be installed on every machine with any
type of connection, including IB and OPA.

So I think that it is enough to be part of that suite and don't invent
our own for one specific tool.


Hello Leon,

Sorry but to me that sounds like a weak argument for including RDMA 
functionality
in iproute2. There is already a library for communication over netlink sockets,
namely libnl. Is there functionality that is in iproute2 but not in libnl and
that is needed for the new tool? If so, have you considered to create a new
library for that functionality?


It is not hard to create new tool, the hardest part is to ensure that it is
part of the distributions. Did you count how many months we are trying to
add rdma-core to debian?


I do agree that it is a strange pairing and am not really a fan. However 
at the end of the day it's just a name for a repo/package. If the 
iproute folks are fine to include rdma in their repo/package, great we 
can leverage their code for CLI and other common stuff.


Now if the interface was something like "ip -FlagForRdma ..." I would 
object to that, but the interface is "rdma ... " so from users 
perspective it's different tools. They don't need to care that it was 
sourced from a common git repo.


Just as an aside this already works a bit with OPA:

 $ ./rdma link
1/1: hfi1_0/1: ifname NONE cap_mask 0x00410022 lid 0x1 lid_mask_count 0 
link_layer InfiniBand
 phys_state 5: LinkUp rate 100 Gb/sec (4X EDR) sm_lid 0x1 sm_sl 
0 state 4: ACTIVE


Leon I'll get you more feedback and testing, I've just been really 
bogged down this week, sorry.


-Denny

Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Thu, May 04, 2017 at 06:30:27PM +, Bart Van Assche wrote:
> On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:
> > On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:
> > > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > > Following our discussion both in mailing list [1] and at the LPC 2016 
> > > > [2],
> > > > we would like to propose this RDMA tool to be part of iproute2 package
> > > > and finally improve this situation.
> > >
> > > Hello Leon,
> > >
> > > Although I really appreciate your work: can you clarify why you would 
> > > like to
> > > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > > motivation
> > > for adding RDMA functionality to iproute2 in [1].
> >
> > We are planning to reuse the same infrastructure provided by iproute2,
> > like netlink parsing, access to distributions, same CLI and same standards.
> >
> > Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, 
> > HFI-VNIC.
> > Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
> > RDMA.
> >
> > I do expect that iproute2 will be installed on every machine with any
> > type of connection, including IB and OPA.
> >
> > So I think that it is enough to be part of that suite and don't invent
> > our own for one specific tool.
>
> Hello Leon,
>
> Sorry but to me that sounds like a weak argument for including RDMA 
> functionality
> in iproute2. There is already a library for communication over netlink 
> sockets,
> namely libnl. Is there functionality that is in iproute2 but not in libnl and
> that is needed for the new tool? If so, have you considered to create a new
> library for that functionality?

It is not hard to create new tool, the hardest part is to ensure that it is
part of the distributions. Did you count how many months we are trying to
add rdma-core to debian?

I have enough headache with that and don't want another one.

Do you have situation in mind where you will have RDMA device without
iproute2 installed?

Thanks

>
> Thanks,
>
> Bart.


signature.asc
Description: PGP signature

Re: [RFC iproute2 0/8] RDMA tool

[Bart Van Assche]

On Thu, 2017-05-04 at 21:25 +0300, Leon Romanovsky wrote:
> On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:
> > On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > > Following our discussion both in mailing list [1] and at the LPC 2016 [2],
> > > we would like to propose this RDMA tool to be part of iproute2 package
> > > and finally improve this situation.
> > 
> > Hello Leon,
> > 
> > Although I really appreciate your work: can you clarify why you would like 
> > to
> > add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> > motivation
> > for adding RDMA functionality to iproute2 in [1].
> 
> We are planning to reuse the same infrastructure provided by iproute2,
> like netlink parsing, access to distributions, same CLI and same standards.
> 
> Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, HFI-VNIC.
> Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
> RDMA.
> 
> I do expect that iproute2 will be installed on every machine with any
> type of connection, including IB and OPA.
> 
> So I think that it is enough to be part of that suite and don't invent
> our own for one specific tool.

Hello Leon,

Sorry but to me that sounds like a weak argument for including RDMA 
functionality
in iproute2. There is already a library for communication over netlink sockets,
namely libnl. Is there functionality that is in iproute2 but not in libnl and
that is needed for the new tool? If so, have you considered to create a new
library for that functionality?

Thanks,

Bart.

Re: [RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

On Thu, May 04, 2017 at 06:10:54PM +, Bart Van Assche wrote:
> On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> > Following our discussion both in mailing list [1] and at the LPC 2016 [2],
> > we would like to propose this RDMA tool to be part of iproute2 package
> > and finally improve this situation.
>
> Hello Leon,
>
> Although I really appreciate your work: can you clarify why you would like to
> add *RDMA* functionality to an *IP routing* tool? I haven't found any 
> motivation
> for adding RDMA functionality to iproute2 in [1].

We are planning to reuse the same infrastructure provided by iproute2,
like netlink parsing, access to distributions, same CLI and same standards.

Right now, RDMA is already tightened to netdev: iWARP, RoCE, IPoIB, HFI-VNIC.
Many drivers (mlx, qed, i40, cxgb) are sharing code between net and
RDMA.

I do expect that iproute2 will be installed on every machine with any
type of connection, including IB and OPA.

So I think that it is enough to be part of that suite and don't invent
our own for one specific tool.

Thanks

>
> Thanks,
>
> Bart.--
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


signature.asc
Description: PGP signature

Re: [RFC iproute2 0/8] RDMA tool

[Bart Van Assche]

On Thu, 2017-05-04 at 21:02 +0300, Leon Romanovsky wrote:
> Following our discussion both in mailing list [1] and at the LPC 2016 [2],
> we would like to propose this RDMA tool to be part of iproute2 package
> and finally improve this situation.

Hello Leon,

Although I really appreciate your work: can you clarify why you would like to
add *RDMA* functionality to an *IP routing* tool? I haven't found any motivation
for adding RDMA functionality to iproute2 in [1].

Thanks,

Bart.

[RFC iproute2 0/8] RDMA tool

[Leon Romanovsky]

From: Leon Romanovsky 


This is initial phase to understand if user experience for this tool fits
RDMA and netdev communities exepectations. Also I would like to get feedback
if it is really worth to provide legacy sysfs for old kernels, or maybe I should
implement netlink from the beginning and abandon sysfs completely.
-

Hi,

Please find below, the patch set with initial implementation of configuration
tool for RDMA subsystem, which will be supplementary tool to already existed
tools in netdev community (ip, devlink, ethtool, ..).

In opposite to netdev community, where standard tools exist to configure and
present different devices abilities, RDMA subsystem historically lacked it.

Following our discussion both in mailing list [1] and at the LPC 2016 [2],
we would like to propose this RDMA tool to be part of iproute2 package
and finally improve this situation.

The development of tool was influenced by ip and devlink tools. This implies
to the object->command interface and naming convention.

In order to close object model, ensure reuse of existing code and make this
tool usable from day one, we decided to implement wrappers over legacy sysfs
prior to implementing netlink functionality. As a nice bonus, it will allow
to use this tool with old kernels too.

It is important to mention that any future extension will be required to be
done with netlink, so for already existing objects small conversion to netlink
will be unavoidable.

  # rdma -h
  Usage: rdma [ OPTIONS ] OBJECT { COMMAND | help }
  where  OBJECT := { dev | link | ipoib | memory | stats | protocols | 
providers | monitor }
  OPTIONS := { -V[ersion] }

* DEV object equals to CA in IBTA specification and will provide
  a way to configure/present settings relevant to specific struct ib_device.

* LINK object represents port in IBTA specification and will give access to
  struct ib_port_immutable. From the day one, It prints netdev name of
  the corresponding IB port that makes ibdev2netdev script redundant.

* IPoIB object is supposed to be specific for IP-over-Infiniband upper
  layer protocol [3]. This ULP was mainly configured by combination
  of various sysfs knobs together with ethtool. Such situation adds
  challenges to add new and expose old configuration settings due to
  the mix between different subsystems.

* MEMORY object will be used to configure memory related settings,
  e.g. on-demand-paging (ODP), force-mr (force usage of MRs for
  RDMA READ/WRITE operation).

* STATS object is needed for everything related to statistics
  (per-PID, per-QP, per-device etc.). Despite the fact that RDMA
  devices provide extensive set of counters, the decision was to
  implement it in netlink directly, because there is a need to add
  filter mechanism to them, which doesn't exist now.

* PROTOCOLS object is going to be used for device special treatment
  of global to protocol settings (e.g. set device in RoCEv2 mode as
  a default, instead of RoCEv1, instead of configfs).

* PROVIDERS objects gives ability to get specific to the device
  information, like supported kABI objects [4].

* MONITOR object is needed to debug netlink communication and will
  follow standard functionality, which exists in ip and devlink tools.

There are number of ULPs which are not covered by this tool yet:
 * HFI-VNIC - I have no access to the HW and believe that Intel
   will add native object support for it.
 * Other storage related ULPs (iSER and SRP) were not introduced too,
   because they have special tools (scci-target-utils) to configure them.
   However it will be pretty straightforward to introduce new object,
   if there is demand for it.

At the initial stage, we implemented infrastructure to read legacy
sysfs entries (Patch #1), initial man pages (Patch #7) and provided
future object examples (Patch #2-6) to allow parallel development.

Following patches will focus on cleaning user interface, parsing other
relevant entries in similar fashion to the link capability mask (Patch #8)
and providing netlink interface.

These patches were tested with two following setups:
 * Setup A:
   - Two Mellanox ConnectX-4 devices (one port)
   - One Mellanox Connect-IB device (two ports)
 * Setup B:
   - One Mellanox ConnectX-4 device (one port)
   - One Mellanox ConnectX-3 Pro device (two ports)

Please consider the inclusion of the RDMA tool into iproute2 package,
so other participants will be able to speed up development.

[1] https://www.mail-archive.com/netdev@vger.kernel.org/msg148523.html
[2] http://www.medkio.com/talks/lpc_debug.pdf
[3] https://tools.ietf.org/html/rfc4392
[4] http://marc.info/?l=linux-rdma=149261526916544=2

TODO: Add json output

Cc: Stephen Hemminger 
Cc: Doug Ledford 
Cc: Jiri Pirko 
Cc: Ariel Almog 
Cc: Dennis Dalessandro 
Cc: Ram Amrani 
Cc: Bart Van Assche