Re: svn commit: r266757 - in head/sys: conf dev/cxgbe dev/cxgbe/common modules/cxgbe/if_cxgbe

2014-05-28 Thread Bjoern A. Zeeb 

On 27 May 2014, at 18:18 , Navdeep Parhar  wrote:

> Author: np
> Date: Tue May 27 18:18:41 2014
> New Revision: 266757
> URL: http://svnweb.freebsd.org/changeset/base/266757
> 
> Log:
>  cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards.
>  Netmap gets its own hardware-assisted virtual interface and won't take
>  over or disrupt the "normal" interface in any way.  You can use both
>  simultaneously.
> 
>  For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl interface
>  (note the 'n' prefix) in the hardware to accompany each cxl
>  interface.  These two ifnet's per port share the same wire but really
>  are separate interfaces in the hardware and software.  Each gets its own
>  L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc.  You
>  should run netmap on the 'n' interfaces only, that's what they are for.
> 
>  With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port
>  of a T580 card.  2 port tx is at ~56Mpps total (28M + 28M) as of now.
>  Single port receive is at 33Mpps but this is very much a work in
>  progress.  I expect it to be closer to 40Mpps once done.  In any case
>  the current effort can already saturate multiple 10G ports of a T5 card
>  at the smallest legal packet size.  T4 gear is totally untested.
> 
>  trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef
>  881.952141 main [1621] interface is ncxl0
>  881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0
>  881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0
>  881.962540 main [1804] mapped 334980KB at 0x801dff000
>  Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus.
>  10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef)
>  881.962562 main [1882] Sending 512 packets every  0.0 s
>  881.962563 main [1884] Wait 2 secs for phy reset
>  884.088516 main [1886] Ready...
>  884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1
>  884.088607 sender_body [996] start
>  884.093246 sender_body [1064] drop copy
>  885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec)
>  886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec)
>  887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec)
>  888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec)
>  889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec)
>  890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec)
>  891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec)
>  892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec)
>  893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec)
>  894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec)
>  895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec)
>  896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec)
>  ...
> 
>  Relnotes:Yes
>  Sponsored by:Chelsio Communications.


Really nice :-)   Thanks a lot for getting it into the tree.


— 
Bjoern A. Zeeb "Come on. Learn, goddamn it.", WarGames, 1983

___
svn-src-head@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r266757 - in head/sys: conf dev/cxgbe dev/cxgbe/common modules/cxgbe/if_cxgbe

2014-05-27 Thread George Neville-Neil 
On 27 May 2014, at 20:42, Navdeep Parhar wrote:

> On 05/27/14 17:36, Luigi Rizzo wrote:
>>
>>
>>
>> On Tue, May 27, 2014 at 8:18 PM, Navdeep Parhar > > wrote:
>>
>>  Author: np
>>  Date: Tue May 27 18:18:41 2014
>>  New Revision: 266757
>>  URL: http://svnweb.freebsd.org/changeset/base/266757
>>
>>  Log:
>>cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards.
>>Netmap gets its own hardware-assisted virtual interface and won't take
>>over or disrupt the "normal" interface in any way.  You can use both
>>simultaneously.
>>
>>For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl interface
>>(note the 'n' prefix) in the hardware to accompany each cxl
>>interface.  These two ifnet's per port share the same wire but really
>>are separate interfaces in the hardware and software.  Each gets
>>  its own
>>L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc.
>>   You
>>should run netmap on the 'n' interfaces only, that's what they are
>>  for.
>>
>>With this, pkt-gen is able to transmit > 45Mpps out of a single
>>  40G port
>>
>>
>> ​and just for the records, this is with 1 core in userland.
>> Very cool.
>>
>> Do you have an easy way to check what kind of throughput
>> you get when the netmap interface has one hw queue ?
>
> Both tx and rx are at around ~33Mpps right now with one hardware queue.
> That's more than enough to saturate a 40G link with 128B sized frames.
>

We need to port more tools to netmap now that it's in more drivers.  I'm
looking at iperf3 but there are other packet oriented daemons we ought
to be porting as well.  PTPd is, of course, on my list.

Best,
George
 
___
svn-src-head@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r266757 - in head/sys: conf dev/cxgbe dev/cxgbe/common modules/cxgbe/if_cxgbe

2014-05-27 Thread Navdeep Parhar 
On 05/27/14 17:36, Luigi Rizzo wrote:
> 
> 
> 
> On Tue, May 27, 2014 at 8:18 PM, Navdeep Parhar  > wrote:
> 
> Author: np
> Date: Tue May 27 18:18:41 2014
> New Revision: 266757
> URL: http://svnweb.freebsd.org/changeset/base/266757
> 
> Log:
>   cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards.
>   Netmap gets its own hardware-assisted virtual interface and won't take
>   over or disrupt the "normal" interface in any way.  You can use both
>   simultaneously.
> 
>   For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl interface
>   (note the 'n' prefix) in the hardware to accompany each cxl
>   interface.  These two ifnet's per port share the same wire but really
>   are separate interfaces in the hardware and software.  Each gets
> its own
>   L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc.
>  You
>   should run netmap on the 'n' interfaces only, that's what they are
> for.
> 
>   With this, pkt-gen is able to transmit > 45Mpps out of a single
> 40G port
> 
> 
> ​and just for the records, this is with 1 core in userland.
> Very cool.
> 
> Do you have an easy way to check what kind of throughput
> you get when the netmap interface has one hw queue ?

Both tx and rx are at around ~33Mpps right now with one hardware queue.
 That's more than enough to saturate a 40G link with 128B sized frames.

Regards,
Navdeep

> 
> cheers
> luigi
> 
>   of a T580 card.  2 port tx is at ~56Mpps total (28M + 28M) as of now.
>   Single port receive is at 33Mpps but this is very much a work in
>   progress.  I expect it to be closer to 40Mpps once done.  In any case
>   the current effort can already saturate multiple 10G ports of a T5
> card
>   at the smallest legal packet size.  T4 gear is totally untested.
> 
>   trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef
>   881.952141 main [1621] interface is ncxl0
>   881.952250 extract_ip_range [275] range is 10.0.0.1:0
>  to 10.0.0.1:0 
>   881.952253 extract_ip_range [275] range is 10.1.0.1:0
>  to 10.1.0.1:0 
>   881.962540 main [1804] mapped 334980KB at 0x801dff000
>   Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus.
>   10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef)
>   881.962562 main [1882] Sending 512 packets every  0.0 s
>   881.962563 main [1884] Wait 2 secs for phy reset
>   884.088516 main [1886] Ready...
>   884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1
>   884.088607 sender_body [996] start
>   884.093246 sender_body [1064] drop copy
>   885.090435 main_thread [1418] 45206353 pps (45289533 pkts in
> 1001840 usec)
>   886.091600 main_thread [1418] 45322792 pps (45375593 pkts in
> 1001165 usec)
>   887.092435 main_thread [1418] 45313992 pps (45351784 pkts in
> 1000834 usec)
>   888.094434 main_thread [1418] 45315765 pps (45406397 pkts in
> 1002000 usec)
>   889.095434 main_thread [1418] 45333218 pps (45378551 pkts in
> 1001000 usec)
>   890.097434 main_thread [1418] 45315247 pps (45405877 pkts in
> 1002000 usec)
>   891.099434 main_thread [1418] 45326515 pps (45417168 pkts in
> 1002000 usec)
>   892.101434 main_thread [1418] 45333039 pps (45423705 pkts in
> 1002000 usec)
>   893.103434 main_thread [1418] 45324105 pps (45414708 pkts in
> 1001999 usec)
>   894.105434 main_thread [1418] 45318042 pps (45408723 pkts in
> 1002001 usec)
>   895.106434 main_thread [1418] 45332430 pps (45377762 pkts in
> 1001000 usec)
>   896.107434 main_thread [1418] 45338072 pps (45383410 pkts in
> 1001000 usec)
>   ...
> 
>   Relnotes: Yes
>   Sponsored by: Chelsio Communications.
> 
> Added:
>   head/sys/dev/cxgbe/t4_netmap.c   (contents, props changed)
> Modified:
>   head/sys/conf/files
>   head/sys/dev/cxgbe/adapter.h
>   head/sys/dev/cxgbe/common/common.h
>   head/sys/dev/cxgbe/common/t4_hw.c
>   head/sys/dev/cxgbe/t4_main.c
>   head/sys/dev/cxgbe/t4_sge.c
>   head/sys/modules/cxgbe/if_cxgbe/Makefile
> 


___
svn-src-head@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r266757 - in head/sys: conf dev/cxgbe dev/cxgbe/common modules/cxgbe/if_cxgbe

2014-05-27 Thread Luigi Rizzo 
On Tue, May 27, 2014 at 8:18 PM, Navdeep Parhar  wrote:

> Author: np
> Date: Tue May 27 18:18:41 2014
> New Revision: 266757
> URL: http://svnweb.freebsd.org/changeset/base/266757
>
> Log:
>   cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards.
>   Netmap gets its own hardware-assisted virtual interface and won't take
>   over or disrupt the "normal" interface in any way.  You can use both
>   simultaneously.
>
>   For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl interface
>   (note the 'n' prefix) in the hardware to accompany each cxl
>   interface.  These two ifnet's per port share the same wire but really
>   are separate interfaces in the hardware and software.  Each gets its own
>   L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc.  You
>   should run netmap on the 'n' interfaces only, that's what they are for.
>
>   With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port
>

​and just for the records, this is with 1 core in userland.
Very cool.

Do you have an easy way to check what kind of throughput
you get when the netmap interface has one hw queue ?

cheers
luigi

  of a T580 card.  2 port tx is at ~56Mpps total (28M + 28M) as of now.
>   Single port receive is at 33Mpps but this is very much a work in
>   progress.  I expect it to be closer to 40Mpps once done.  In any case
>   the current effort can already saturate multiple 10G ports of a T5 card
>   at the smallest legal packet size.  T4 gear is totally untested.
>
>   trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef
>   881.952141 main [1621] interface is ncxl0
>   881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0
>   881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0
>   881.962540 main [1804] mapped 334980KB at 0x801dff000
>   Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus.
>   10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef)
>   881.962562 main [1882] Sending 512 packets every  0.0 s
>   881.962563 main [1884] Wait 2 secs for phy reset
>   884.088516 main [1886] Ready...
>   884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1
>   884.088607 sender_body [996] start
>   884.093246 sender_body [1064] drop copy
>   885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840
> usec)
>   886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165
> usec)
>   887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834
> usec)
>   888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000
> usec)
>   889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000
> usec)
>   890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000
> usec)
>   891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000
> usec)
>   892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000
> usec)
>   893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999
> usec)
>   894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001
> usec)
>   895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000
> usec)
>   896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000
> usec)
>   ...
>
>   Relnotes: Yes
>   Sponsored by: Chelsio Communications.
>
> Added:
>   head/sys/dev/cxgbe/t4_netmap.c   (contents, props changed)
> Modified:
>   head/sys/conf/files
>   head/sys/dev/cxgbe/adapter.h
>   head/sys/dev/cxgbe/common/common.h
>   head/sys/dev/cxgbe/common/t4_hw.c
>   head/sys/dev/cxgbe/t4_main.c
>   head/sys/dev/cxgbe/t4_sge.c
>   head/sys/modules/cxgbe/if_cxgbe/Makefile
>
>
___
svn-src-head@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

svn commit: r266757 - in head/sys: conf dev/cxgbe dev/cxgbe/common modules/cxgbe/if_cxgbe

2014-05-27 Thread Navdeep Parhar 
Author: np
Date: Tue May 27 18:18:41 2014
New Revision: 266757
URL: http://svnweb.freebsd.org/changeset/base/266757

Log:
  cxgbe(4): netmap support for Terminator 5 (T5) based 10G/40G cards.
  Netmap gets its own hardware-assisted virtual interface and won't take
  over or disrupt the "normal" interface in any way.  You can use both
  simultaneously.
  
  For kernels with DEV_NETMAP, cxgbe(4) carves out an ncxl interface
  (note the 'n' prefix) in the hardware to accompany each cxl
  interface.  These two ifnet's per port share the same wire but really
  are separate interfaces in the hardware and software.  Each gets its own
  L2 MAC addresses (unicast and multicast), MTU, checksum caps, etc.  You
  should run netmap on the 'n' interfaces only, that's what they are for.
  
  With this, pkt-gen is able to transmit > 45Mpps out of a single 40G port
  of a T580 card.  2 port tx is at ~56Mpps total (28M + 28M) as of now.
  Single port receive is at 33Mpps but this is very much a work in
  progress.  I expect it to be closer to 40Mpps once done.  In any case
  the current effort can already saturate multiple 10G ports of a T5 card
  at the smallest legal packet size.  T4 gear is totally untested.
  
  trantor:~# ./pkt-gen -i ncxl0 -f tx -D 00:07:43:ab:cd:ef
  881.952141 main [1621] interface is ncxl0
  881.952250 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0
  881.952253 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0
  881.962540 main [1804] mapped 334980KB at 0x801dff000
  Sending on netmap:ncxl0: 4 queues, 1 threads and 1 cpus.
  10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> 00:07:43:ab:cd:ef)
  881.962562 main [1882] Sending 512 packets every  0.0 s
  881.962563 main [1884] Wait 2 secs for phy reset
  884.088516 main [1886] Ready...
  884.088535 nm_open [457] overriding ifname ncxl0 ringid 0x0 flags 0x1
  884.088607 sender_body [996] start
  884.093246 sender_body [1064] drop copy
  885.090435 main_thread [1418] 45206353 pps (45289533 pkts in 1001840 usec)
  886.091600 main_thread [1418] 45322792 pps (45375593 pkts in 1001165 usec)
  887.092435 main_thread [1418] 45313992 pps (45351784 pkts in 1000834 usec)
  888.094434 main_thread [1418] 45315765 pps (45406397 pkts in 1002000 usec)
  889.095434 main_thread [1418] 45333218 pps (45378551 pkts in 1001000 usec)
  890.097434 main_thread [1418] 45315247 pps (45405877 pkts in 1002000 usec)
  891.099434 main_thread [1418] 45326515 pps (45417168 pkts in 1002000 usec)
  892.101434 main_thread [1418] 45333039 pps (45423705 pkts in 1002000 usec)
  893.103434 main_thread [1418] 45324105 pps (45414708 pkts in 1001999 usec)
  894.105434 main_thread [1418] 45318042 pps (45408723 pkts in 1002001 usec)
  895.106434 main_thread [1418] 45332430 pps (45377762 pkts in 1001000 usec)
  896.107434 main_thread [1418] 45338072 pps (45383410 pkts in 1001000 usec)
  ...
  
  Relnotes: Yes
  Sponsored by: Chelsio Communications.

Added:
  head/sys/dev/cxgbe/t4_netmap.c   (contents, props changed)
Modified:
  head/sys/conf/files
  head/sys/dev/cxgbe/adapter.h
  head/sys/dev/cxgbe/common/common.h
  head/sys/dev/cxgbe/common/t4_hw.c
  head/sys/dev/cxgbe/t4_main.c
  head/sys/dev/cxgbe/t4_sge.c
  head/sys/modules/cxgbe/if_cxgbe/Makefile

Modified: head/sys/conf/files
==
--- head/sys/conf/files Tue May 27 16:30:54 2014(r266756)
+++ head/sys/conf/files Tue May 27 18:18:41 2014(r266757)
@@ -1127,6 +1127,8 @@ dev/cxgb/cxgb_t3fw.c  optional cxgb cxgb
compile-with "${NORMAL_C} -I$S/dev/cxgb"
 dev/cxgbe/t4_main.coptional cxgbe pci \
compile-with "${NORMAL_C} -I$S/dev/cxgbe"
+dev/cxgbe/t4_netmap.c  optional cxgbe pci \
+   compile-with "${NORMAL_C} -I$S/dev/cxgbe"
 dev/cxgbe/t4_sge.c optional cxgbe pci \
compile-with "${NORMAL_C} -I$S/dev/cxgbe"
 dev/cxgbe/t4_l2t.c optional cxgbe pci \

Modified: head/sys/dev/cxgbe/adapter.h
==
--- head/sys/dev/cxgbe/adapter.hTue May 27 16:30:54 2014
(r266756)
+++ head/sys/dev/cxgbe/adapter.hTue May 27 18:18:41 2014
(r266757)
@@ -54,6 +54,7 @@
 #include 
 
 #include "offload.h"
+#include "common/t4_msg.h"
 #include "firmware/t4fw_interface.h"
 
 MALLOC_DECLARE(M_CXGBE);
@@ -131,6 +132,7 @@ enum {
RX_IQ_ESIZE = 64,   /* At least 64 so CPL_RX_PKT will fit */
 
EQ_ESIZE = 64,  /* All egress queues use this entry size */
+   SGE_MAX_WR_NDESC = SGE_MAX_WR_LEN / EQ_ESIZE, /* max WR size in desc */
 
RX_FL_ESIZE = EQ_ESIZE, /* 8 64bit addresses */
 #if MJUMPAGESIZE != MCLBYTES
@@ -155,6 +157,17 @@ enum {
 };
 
 enum {
+   XGMAC_MTU   = (1 << 0),
+   XGMAC_PROMISC   = (1 << 1),
+   XGMAC_ALLMULTI  = (1 << 2),
+   XGMAC_VLANEX= (1 << 3),
+   XGMAC_UCADDR= (1 << 4),
+   XGMAC_MCADDRS   = (1